Lessons learned in systems change initiatives: benchmarks and indicators

Developing and Evaluating a Data-Driven and Systems Approach to Health Promotion Among Vocational Students: Protocol for the Data Health Study

BACKGROUND

Vocational school students exhibit significant risk behaviors in terms of poor diet, frequent use of nicotine products, inadequate fruit and vegetable intake, low levels of physical activity, and poor mental health. This makes vocational students vulnerable to the development of noncommunicable diseases. Therefore, effective health promotion programs targeting vocational students are required.

OBJECTIVE

The Danish study "Data-driven and Systems Approach to Health Promotion Among Vocational Students" (Data Health) aims to develop, implement, and evaluate a systems approach to support vocational schools, municipalities, and local communities in implementing locally relevant health promotion actions among and for vocational students. This paper describes the Data Health program and how implementation and preliminary effectiveness will be evaluated.

METHODS

The Data Health program offers an iterative 5-step process to develop changes in the systems that shape health behavior and well-being among vocational students. The program will be implemented and evaluated in 8 Danish vocational schools in 4 municipalities. The implementation of the process and actions will be explored using a systems-based evaluation design that assesses contextual differences and the mechanisms through which the program leads to changes in the systems. Preliminary effectiveness at the individual level (students' self-reported health behavior and well-being) and organizational level (school organizational readiness reported by school staff) will be assessed using a quasi-experimental design, and cross-sectional data will be collected at all 8 schools simultaneously 4 times during the 2-year study period.

RESULTS

This study was launched in 2021, and data collection is expected to be completed in June 2024. The first results are expected to be submitted for publication in January 2024.

CONCLUSIONS

We expect that the Data Health study will make significant contributions to complex intervention research by contributing to the paucity of research studies that have used systems approaches in school settings. The study will also provide evidence of successful elements for systems change and effectiveness to determine whether a national scale-up can be recommended.

TRIAL REGISTRATION

ClinicalTrials.gov NCT05308459; https://clinicaltrials.gov/study/NCT05308459.

INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID)

DERR1-10.2196/52571.

Incorporating multiple data sources to assess changes in organizational capacity in child welfare systems

Three federally funded Child Welfare Capacity Building Centers provide services to build the organizational capacity of public child welfare agencies to help meet federal requirements, improve practice, and improve outcomes for children and families. The aim of this study was to explore capacity outcomes in five dimensions - resources, infrastructure, knowledge and skills, culture and climate, and engagement and partnership - achieved by child welfare jurisdictions who received Center services. Analyses describe the capacities targeted for improvement and the amount and type of services provided by Centers; assess the relationship between services and capacity increase; and explore whether that relationship differs depending on the jurisdiction's level of foundational capacity. Data collected through surveys and a service delivery tracking system reflect the perspectives of service recipients and service providers. Results reveal jurisdictions typically targeted capacity outcomes in the dimensions of knowledge and skills and infrastructure and received an average of 28 hours of direct services to support their capacity-building efforts. Dosage of service was positively associated with achievement of capacity outcomes, though no interaction was found between service dosage and foundational capacity in the effect on outcomes. Methodological lessons learned and implications for future evaluations of organizational capacity building efforts are offered.

How We Get Free: Graduate Training as an Opportunity for Equitable Participation and Liberation

In this conceptual article, we assert that psychology should be transformed to adopt the explicit goal of working toward the liberation of people oppressed by society rather than striving for mere equality. To achieve such a transformation, it is necessary to reenvision graduate training in psychology. Graduate training in psychology is an important vehicle by which psychologists can become prepared to use research and practice to eradicate inequities in society. Therefore, we propose six pillars for liberation-focused graduate training in psychology: critical unlearning/unknowing, cooperative modes of production, prioritizing indigenous knowledge, embedded interdependence, systems-level action, and prioritizing members of oppressed groups. Although this conceptualization may engender resistance, we argue that there are many potential pathways by which graduate training may use liberation psychology to work equitably with oppressed groups to seek justice.

"I'm No Criminal, I'm Just Homeless": The Greensboro Homeless Union's efforts to address the criminalization of homelessness

This paper examines how homelessness is criminalized in Greensboro, North Carolina, and the ways in which the Homeless Union of Greensboro (HUG) has contested such criminalization. This paper draws on data from a participatory action research study conducted between 2018 and 2020 by a group of researchers from two local universities and members of HUG. Findings from our study suggest that law enforcement officers in Greensboro use a vast array of laws to harass, ticket, and arrest people experiencing homelessness, particularly those who are Black. Findings also suggest that when individuals experiencing homelessness seek help for citations or arrests, it is challenging to access quality, affordable legal representation. This paper illustrates how HUG takes a multi-pronged approach to address the variety of policies and practices that target homeless people, particularly people of color, recognizing that systems change requires a multifaceted approach that adapts to dynamic social and political contexts.

The ENCOMPASS framework: a practical guide for the evaluation of public health programmes in complex adaptive systems

Background

Systems thinking embraces the complexity of public health problems, including childhood overweight and obesity. It aids in understanding how factors are interrelated, and it can be targeted to produce favourable changes in a system. There is a growing call for systems approaches in public health research, yet limited practical guidance is available on how to evaluate public health programmes within complex adaptive systems. The aim of this paper is to present an evaluation framework that supports researchers in designing systems evaluations in a comprehensive and practical way.

Methods

We searched the literature for existing public health systems evaluation studies. Key characteristics on how to conduct a systems evaluation were extracted and compared across studies. Next, we overlaid the identified characteristics to the context of the Lifestyle Innovations Based on Youth Knowledge and Experience (LIKE) programme evaluation and analyzed which characteristics were essential to carry out the LIKE evaluation. This resulted in the Evaluation of Programmes in Complex Adaptive Systems (ENCOMPASS) framework.

Results

The ENCOMPASS framework includes five iterative stages: (1) adopting a system dynamics perspective on the overall evaluation design; (2) defining the system boundaries; (3) understanding the pre-existing system to inform system changes; (4) monitoring dynamic programme output at different system levels; and (5) measuring programme outcome and impact in terms of system changes.

Conclusions

The value of ENCOMPASS lies in the integration of key characteristics from existing systems evaluation studies, as well as in its practical, applied focus. It can be employed in evaluating public health programmes in complex adaptive systems. Furthermore, ENCOMPASS provides guidance for the entire evaluation process, all the way from understanding the system to developing actions to change it and to measuring system changes. By the nature of systems thinking, the ENCOMPASS framework will likely evolve further over time, as the field expands with more completed studies.

Providing Coordinated, Immediate, Trauma-Focused, and Interdisciplinary Responses to Children Exposed to Severe Intimate Partner Violence: Assessing Feasibility of a Collaborative Model

Despite the known consequences associated with children's exposure to intimate partner violence (IPV), numerous children exposed to IPV never access victim services and/or mental health treatment. Informed by the Child Development-Community Policing Program (CD-CP), the Child Trauma Response Team (CTRT) is designed to provide a coordinated, immediate, trauma-informed, and interdisciplinary response to children, adolescents, and their impacted family members who are exposed to severe IPV. The aim of this study was to explore the feasibility of the CTRT from the perspective of the key stakeholders and to identify what facilitated and/or hindered the collaborative model. Interviews were conducted with all 12 CTRT stakeholders of whom three worked for law enforcement, four worked for the district attorney's office, four worked for the nonprofit victim service organization, and one worked for the city office funding the pilot. Results indicated that the CTRT program evolved through a cyclical process including (a) implementing and enhancing program services, (b) defining and nurturing partner relationships, and (c) shifting practices among partnering agencies. The results provide concrete tools and practices that were successful in the CTRT pilot implementation. Despite the potential barriers to successful multidisciplinary collaborations, it is critical that we invest in developing and implementing collaborative intervention models in the IPV field. The CTRT model is an innovative model that conducts multidisciplinary coordinated outreach to families experiencing severe IPV to enhance family engagement in services and, when appropriate, in the criminal justice process.

Services for Adults With Autism Spectrum Disorder: a Systems Perspective

PURPOSE OF REVIEW

We review original research about services for adults on the autism spectrum published from January 2013 through December 2018. The main aim is to characterize the topical and methodological aspects of research about services. We review research on services related to employment, living in the community, and social participation. We compare our results with those from a similar review published in 2012 to assess progress and identify where new directions in research about services for adults with autism are needed.

RECENT FINDINGS

We found the evidence base about services for adults on the autism spectrum remains very small and highly variable in aims and methods. There is wide variability in methods used to define sampling frames and recruit participants. Most studies focus on employment. Almost no studies examine the overall ecosystem of services serving autistic adults. Few studies use a conceptual framework for understanding access to, or improvement of, services. The small size of the extant research coupled with inconsistent quality prevents the accumulation of new knowledge in ways that would significantly inform the improvement of systems of care for the growing population of adults on the autism spectrum.

Defining and Measuring Sustainability in the Food & Fitness Initiative

Several frameworks for defining and measuring sustainability in public health have been documented in the literature. For the Food & Fitness Initiative, sustainability was a central aim at the outset and was defined broadly throughout the project. Sustainability included not only funding and resources necessary to support organizational structures but was a core function of how these partnerships were able to focus their work, build capacity, forge lasting relationships, execute the work, and produce systems and policy changes that would endure over time. In this article, we present findings from an online survey assessing partners' views on 10 distinct dimensions of sustainability and several key themes from a set of key informant interviews with partnership leaders. Of the 10 dimensions, participants reported having the most success in creating (1) community ownership, where initiatives are led by and reflect the needs of community residents; (2) a vision that is shared across partners and developed collaboratively; and (3) leadership that includes a diverse team of skilled, credible people. A key learning in this project was that sustainability needs to be intentional and clearly defined and that evaluations should include multiple and ongoing methods to capture several dimensions of sustainability.

Crossing the Age Divide: Cross-Age Collaboration Between Programs Serving Transition-Age Youth

Programs that serve transition-age youth with serious mental health conditions typically reside in either the child or the adult system. Good service provision calls for interactions among these programs. The objective of this research was to discover programmatic characteristics that facilitate or impede collaboration with programs serving dissimilar age groups, among programs that serve transition-age youth. To examine this "cross-age collaboration," this research used social network analysis methods to generate homophily and heterophily scores in three communities that had received federal grants to improve services for this population. Heterophily scores (i.e., a measure of cross-age collaboration) in programs serving only transition-age youth were significantly higher than the heterophily scores of programs that served only adults or only children. Few other program markers or malleable program factors predicted heterophily. Programs that specialize in serving transition-age youth are a good resource for gaining knowledge of how to bridge adult and child programs.

Community and state systems change associated with the healthy transitions initiative

People engaged in efforts to improve services to emerging adults with serious mental health challenges have reached the conclusion that service change at the program or agency level is not sustainable without related changes at the systems or policy level. This article focuses on one set of efforts to create intentional system change at both the community and state levels. These changes were pursued by states and communities that received grants under the federally funded Healthy Transitions Initiative (HTI), with the aim of creating more effective services for emerging adults with serious mental health conditions. The article reviews the development of a measure to assess systems change efforts at the state and community levels and describes the findings that emerged when the measure was used to assess the change that occurred in the HTI sites over a period of approximately three and a half years.

Development of a facilitation curriculum to support primary care transformation: the "coach medical home" curriculum

BACKGROUND

In an effort to improve patient care, retain high-quality primary care providers, and control costs, primary care practices across the United States are transforming to patient-centered medical homes. This is no small task. Practice facilitation, also called "coaching," is increasingly being used to support system change; however, there is limited guidance for these programs.

OBJECTIVE

To develop an evidence-based curriculum to help practice coaches guide broad-scale transformation efforts in primary care.

METHODS

We gathered evidence about effective practice transformation coaching from 25 published programs and 8 expert interviews. Given limited published information, we drew extensively on our experience as leaders and coaches in the Safety Net Medical Home Initiative. Using these data, and with input from a User Group, we identified 6 curricular topics and created learning objectives and curricular content related to these topics.

RESULTS

The Coach Medical Home curriculum guides coaches in the following areas: getting started with a practice; recognition and payment; sequencing changes; measurement; learning communities; and sustainability and spread.

CONCLUSIONS

Coach Medical Home is a publically available web-based curriculum that provides tools, resources, and guidance for practice transformation support programs, including practice facilitators and learning community organizers.

Designing a theory-informed, contextually appropriate intervention strategy to improve delivery of paediatric services in Kenyan hospitals

BACKGROUND

District hospital services in Kenya and many low-income countries should deliver proven, effective interventions that could substantially reduce child and newborn mortality. However such services are often of poor quality. Researchers have therefore been challenged to identify intervention strategies that go beyond addressing knowledge, skill, or resource inadequacies to support health systems to deliver better services at scale. An effort to develop a system-oriented intervention tailored to local needs and context and drawing on theory is described.

METHODS

An intervention was designed to improve district hospital services for children based on four main strategies: a reflective process to distill root causes for the observed problems with service delivery; developing a set of possible intervention approaches to address these problems; a search of literature for theory that provided the most appropriate basis for intervention design; and repeatedly moving backwards and forwards between identified causes, proposed interventions, identified theory, and knowledge of the existing context to develop an overarching intervention that seemed feasible and likely to be acceptable and potentially sustainable.

RESULTS AND DISCUSSION

In addition to human and resource constraints key problems included failures of relevant professionals to take responsibility for or ownership of the challenge of pediatric service delivery; inadequately prepared, poorly supported leaders of service units (mid-level managers) who are often professionally and geographically isolated and an almost complete lack of useful information for routinely monitoring or understanding service delivery practice or outcomes. A system-oriented intervention recognizing the pivotal role of leaders of service units but addressing the outer and inner setting of hospitals was designed to help shape and support an appropriate role for these professionals. It aims to foster a sense of ownership while providing the necessary understanding, knowledge, and skills for mid-level managers to work effectively with senior managers and frontline staff to improve services. The intervention will include development of an information system, feedback mechanisms, and discussion fora that promote positive change. The vehicle for such an intervention is a collaborative network partnering government and national professional associations. This case is presented to promote discussion on approaches to developing context appropriate interventions particularly in international health.

Accomplishing structural change: identifying intermediate indicators of success

Coalitions are routinely employed across the United States as a method of mobilizing communities to improve local conditions that impact on citizens' well-being. Success in achieving specific objectives for environmental or structural community change may not quickly translate into improved population outcomes in the community, posing a dilemma for coalitions that pursue changes that focus on altering community conditions. Considerable effort by communities to plan for and pursue structural change objectives, without evidence of logical and appropriate intermediate markers of success could lead to wasted effort. Yet, the current literature provides little guidance on how coalitions might select intermediate indicators of achievement to judge their progress and the utility of their effort. The current paper explores the strengths and weaknesses of various indicators of intermediate success in creating structural changes among a sample of 13 coalitions organized to prevent exposure to HIV among high-risk adolescents in their local communities.

Evaluating environmental change strategies: challenges and solutions

In this introductory article we define environmental change strategies (ECS), summarize the primary challenges associated with evaluating ECS, and provide an overview of the methods researchers have employed to begin to address these challenges. This special issue provides a range of examples, from researchers and practitioners in the field, of different approaches for addressing these challenges. These articles present new methods to understand and test how ECS are implemented and propose methods to evaluate their implementation. The content of the articles covers multiple public health issues, including substance abuse prevention, tobacco control, HIV prevention, and obesity prevention. This special issue is intended to build the evidence base for effective ECS, generate compelling discussion, critical analyses, and spur future research that will help improve the implementation and evaluation of ECS.

Implementation and development of federally-funded systems of care over time

This study describes development in federally funded systems of care. Data for this study were collected using the System of Care Assessment that rated grantees' enactment of system of care principles in the infrastructure and service delivery domains. Data were collected by trained raters who conducted several site visits over the funding period. This study described system development over time across 61 sites and tested whether gains were statistically significant. Latent profile analysis was used to explore whether sites could be meaningfully grouped based on their baseline service delivery domain scores. Differences across groups were tested in terms of community, system, and client characteristics. Differential growth across groups was also examined. Overall, systems of care developed over time in both the infrastructure and service delivery domains. Although infrastructure scores were generally lower than service delivery scores, greater gains were seen for the infrastructure domain. Three groups of sites were identified that could be distinguished in terms of degree of development overall, and for specific system of care principles. The groups of sites differed in terms of community factors, system features, and client characteristics. In addition, repeated measures analyses found differential growth in system development over time across groups.

Creating an environmental justice framework for policy change in childhood asthma: a grassroots to treetops approach

OBJECTIVES

The Community Action to Fight Asthma Initiative, a network of coalitions and technical assistance providers in California, employed an environmental justice approach to reduce risk factors for asthma in school-aged children. Policy advocacy focused on housing, schools, and outdoor air quality. Technical assistance partners from environmental science, policy advocacy, asthma prevention, and media assisted in advocacy. An evaluation team assessed progress and outcomes.

METHODS

A theory of change and corresponding logic model were used to document coalition development and successes. Site visits, surveys, policymaker interviews, and participation in meetings documented the processes and outcomes. Quantitative and qualitative data were analyzed to assess strategies, successes, and challenges.

RESULTS

Coalitions, working with community residents and technical assistance experts, successfully advocated for policies to reduce children's exposures to environmental triggers, particularly in low-income communities and communities of color. Policies were implemented at various levels.

CONCLUSIONS

Environmental justice approaches to policy advocacy could be an effective strategy to address inequities across communities. Strong technical assistance, close community involvement, and multilevel strategies were all essential to effective policies to reduce environmental inequities.

Theorising interventions as events in systems

Conventional thinking about preventive interventions focuses over simplistically on the "package" of activities and/or their educational messages. An alternative is to focus on the dynamic properties of the context into which the intervention is introduced. Schools, communities and worksites can be thought of as complex ecological systems. They can be theorised on three dimensions: (1) their constituent activity settings (e.g., clubs, festivals, assemblies, classrooms); (2) the social networks that connect the people and the settings; and (3) time. An intervention may then be seen as a critical event in the history of a system, leading to the evolution of new structures of interaction and new shared meanings. Interventions impact on evolving networks of person-time-place interaction, changing relationships, displacing existing activities and redistributing and transforming resources. This alternative view has significant implications for how interventions should be evaluated and how they could be made more effective. We explore this idea, drawing on social network analysis and complex systems theory.

Community psychology: individuals and interventions in community context

Community psychology has historically focused on understanding individual behavior in sociocultural context, assessing high-impact contexts, and working in and with communities to improve their resources and influence over their futures. This review adopts an ecological perspective on recent developments in the field, beginning with philosophy of science and progressing through a series of substantive research and intervention domains that characterize current work. These domains include research on the ecology of lives, the assessment of social settings and their impact on behavior, culture and diversity as expressed in the community research process, and community intervention.

Collaborative efforts to improve system response to families who are experiencing child maltreatment and domestic violence

The Greenbook demonstration initiative provided federal funding and other support to six communities to establish collaborations to plan and implement policy and practice changes in systems that serve families who are experiencing domestic violence and child maltreatment or child exposure to domestic violence. The demonstration sites established and organized collaborative groups in accordance with the Greenbook foundational principles and recommendations, including representation from multiple levels within the primary partner systems and other organizations, as well as the community. The sites struggled with how to engage consumers of the primary systems, however, and devoted a great deal of time to understanding and addressing organizational differences among the partners. Other salient collaborative influences included leadership, resources, trust, and commitment. The stakeholders noted that the collaborative relationships required a great deal of work but were ultimately one of the main successes of the initiative.