Exploring the impact of evaluation on learning and health innovation sustainability: protocol for a realist synthesis

BACKGROUND

Within the Learning Health System (LHS) model, learning routines, including evaluation, allow for continuous incremental change to take place. Within these learning routines, evaluation assists in problem identification, data collection, and data transformation into contextualized information, which is then re-applied to the LHS environment. Evaluation that catalyzes learning and improvement may also contribute to health innovation sustainability. However, there is little consensus as to why certain evaluations seem to support learning and sustainability, while others impede it. This realist synthesis seeks to understand the contextual factors and underlying mechanisms or drivers that best support health systems learning and sustainable innovation.

METHODS

This synthesis will be guided by Pawson and colleagues' 2005 and Emmel and colleagues' 2018 guidelines for conducting realist syntheses. The review process will encompass five steps: (1) scoping the review, (2) building theories, (3) identifying the evidence, (4) evidence selection and appraisal, and (5) data extraction and synthesis. An Expert Committee comprised of leaders in evaluation, innovation, sustainability, and realist methodology will guide this synthesis. Review findings will be reported using the RAMESES guidelines.

DISCUSSION

The use of a realist review will allow for exploration and theorizing about the contextual factors and underlying mechanisms that make evaluations 'work' (or 'not work') to support learning and sustainability. Depending on results, we will attempt to synthesize findings into a series of recommendations for evaluations with the intention to support health systems learning and sustainability. Finalized results will be presented at national and international conferences, as well as disseminated via a peer-reviewed publication.

SYSTEMATIC REVIEW REGISTRATION

This realist synthesis protocol has been registered with PROSPERO ( https://www.crd.york.ac.uk/prospero/  ID 382690).

Strategies used throughout the scaling-up process of eConsult - Multiple case study of four Canadian Provinces

BACKGROUND

eConsult is a model of asynchronous communication connecting primary care providers to specialists to discuss patient care. This study aims to analyze the scaling-up process and identify strategies used to support scaling-up efforts in four provinces in Canada.

METHODS

We conducted a multiple case study with four cases (ON, QC, MB, NL). Data collection methods included document review (n = 93), meeting observations (n = 65) and semi-structured interviews (n = 40). Each case was analyzed based on Milat's framework.

RESULTS

The first scaling-up phase was marked by the rigorous evaluation of eConsult pilot projects and the publication of over 90 scientific papers. In the second phase, provinces implemented provincial multi-stakeholder committees, institutionalized the evaluation, and produced documents detailing the scaling-up plan. During the third phase, efforts were made to lead proofs of concept, obtain the endorsement of national and provincial organizations, and mobilize alternate sources of funding. The last phase was mainly observed in Ontario, where the creation of a provincial governance structure and strategies were put in place to monitor the service and manage changes.

CONCLUSIONS

Various strategies need to be used throughout the scaling-up process. The process remains challenging and lengthy because health systems lack clear processes to support innovation scaling-up.

Scaling-Up eConsult: Promising Strategies to Address Enabling Factors in Four Jurisdictions in Canada

BACKGROUND

Effective healthcare innovations are often not scaled up beyond their initial local context. Lack of practical knowledge on how to move from local innovations to large-system improvement hinders innovation and learning capacity in health systems. Studying scale-up processes can lead to a better understanding of how to facilitate the scale-up of interventions. eConsult is a digital health innovation that aims to connect primary care professionals with specialists through an asynchronous electronic consultation. The recent implementation of eConsult in the public health systems of four Canadian jurisdictions provides a unique opportunity to identify different enabling strategies and related factors that promote the scaling up of eConsult across jurisdictions.

METHODS

We conducted a narrative case study in four Canadian provinces, Quebec, Ontario, Manitoba, and Newfoundland & Labrador, over a 3-year period (2018-2021). We observed provincial eConsult committee meetings (n=65) and national eConsult forums (n=3), and we reviewed internal documents (n=93). We conducted semi-structured interviews with key actors in each jurisdiction (eg, researchers, primary care professionals, specialists, policy-makers, and patient partners) (n=40). We conducted thematic analysis guided by the literature on factors and strategies used to scale up innovations.

RESULTS

We identified a total of 31 strategies related to six key enabling factors to scaling up eConsult, including: (1) multi-actor engagement; (2) relative advantage; (3) knowledge transfer; (4) strong evidence base; (5) physician leadership; and (6) resource acquisition (eg, human, material, and financial resources). More commonly used strategies, such as leveraging research infrastructure and bringing together various actors, were used to address multiple enabling factors.

CONCLUSION

Actors used various strategies to scale up eConsult within their respective contexts, and these helped address six key factors that seemed to be essential to the scale-up of eConsult.

Mapping the Newcomer Journey for More Equitable Population Health: Insights from an Ontario Health Team

The COVID-19 pandemic has magnified systemic vulnerabilities and made the global and Canadian newcomer experience even more fragile. In 2022, the Kitchener, Waterloo, Wellesley, Wilmot and Woolwich (KW4) Ontario Health Team launched a journey-mapping initiative with the aim to better understand newcomers' lived experiences with regard to their health and wellness within the first two years of their arrival in the region. We interviewed 17 newcomers from 11 different countries. The outcomes of this project are helping to inform a people-centred integrated health system approach toward service redesign and the creation of technological solutions to improve newcomers' abilities to self-navigate local services toward more equitable population health outcomes.

How Openness Serves Innovation in Healthcare? Comment on "What Managers Find Important for Implementation of Innovations in the Healthcare Sector - Practice Through Six Management Perspectives"

The recent study of which enabling factors can facilitate the specific step of moving from idea generation to implementation in healthcare supports that managing innovation is a context-driven process that goes through six categories of change. While this research provides a general and rather comprehensives overview of what successful innovation work needs, it does not offer deeper insights into how categories of change can be operated in the context of accelerated openness in healthcare. I use the concepts of open innovation and open strategy to trying better understand how openness, in terms of greater inclusion and transparency, may or may not serve healthcare innovation through three theoretical questions: to whom, how and when to open up to foster innovation? Whilst diversity of knowledge, actors and systems are growing drivers of innovation, strategizing openness for more deliberate and impactful inclusion and transparency in healthcare management is key to coproducing better health.

Appropriating integrated performance management tools in healthcare: a sociomaterial work story

PURPOSE

The purpose of this paper is to explore the appropriation of control rooms based on value-based integrated performance management tools implemented in all publicly funded health organizations in Quebec (Canada) as a form of legitimate sociomaterial work.

DESIGN/METHODOLOGY/APPROACH

Multi-site organizational ethnographic case studies in two Integrated health and social services centers, with narrative process analysis of triangulated qualitative data collected through non-participant observation (163 h), individual semi-structured interviews (N = 34), and document review (N = 143).

FINDINGS

Three types of legitimate sociomaterial work are accomplished when actors appropriate control rooms: 1) reformulating performance management work; 2) disrupting accountability work and; 3) effecting value-based integrated performance management. Each actor (tools, institutions and people) follows recurrent institutional work-paths: tools consistently engage in disruptive work; institutions consistently engage in maintaining work, and people consistently engage in creation work. The study reveals the potential of performance management tools as "effective integrators" of the technological, managerial, policy and delivery levels of data-driven health system performance and improvement.

PRACTICAL IMPLICATIONS

This paper draws on theoretically informed empirical insights to develop actionable knowledge around how to better design, implement and adapt tool-driven health system change: 1) Packaging the three agents of data-driven system change in health care: tools, institutions, people; 2) Redefining the search for performance in health care in the context of value creation, and; 3) Strengthening clinical and managerial relevance in health performance management practice.

ORIGINALITY/VALUE

The authors aim to stimulate new and original scholarship around the under-theorized concept of sociomaterial work, challenging theoretical, ontological and practical conceptions of work in healthcare organizations and beyond.

Control rooms in publicly-funded health systems: Reviving value in healthcare governance

BACKGROUND

As part of reforms in 2015, the Ministry of Health and Social Services in Quebec, Canada mandated the national implementation of control rooms, making health system actors accountable for implementing value-based performance management.

OBJECTIVE

To explore how do organizational actors appropriate control rooms as managerial tools to influence value-based performance in health systems.

DESIGN

Multi-site organizational ethnographic case studies (N = 2) and narrative process analysis of triangulated qualitative data collected through non-participatory observations (179.5 h), individual semi-structured interviews (N = 34), and document review (N = 143).

RESULTS

The process of appropriating control rooms plays a crucial role in achieving value-based performance management. Appropriating unfolds along three paths (cognitive, structural, technical) over three phases (implementing, testing, adapting). Implementing control rooms both produces and emerges from improvement capacities within healthcare organizations. Testing tools reveals that incompatibilities between tools, structures and values give rise to value-driven distributed clinical leadership. Adapting tools relies on the adaptability of organizations towards the value system driving the tools, rather than on the adaptability of tools to organizational design.

CONCLUSION

There is no "one-size-fits-all" framework to design and support the successful appropriation of control rooms towards achieving value-based performance. However, we believe that consideration for the three distinct phases of appropriation and leveraging the right mechanism to support each phase is a first important step in reviving value in healthcare governance.

Lessons from mandated implementation of a performance management system

PURPOSE

Lean-inspired approaches and performance management systems are being implemented in public healthcare organisations internationally. However, the literature is inconclusive regarding the benefits of these management tools and there is a lack of knowledge regarding processes for large-scale implementation of these tools. This article aims to describe the implementation process and to better understand how this process influences the mandated performance management system.

DESIGN/METHODOLOGY/APPROACH

This research is based on a comparative case study of three healthcare organisations in Canada. Data consist documents, non-participant observation and semi-structured interviews with key actors (n = 30). Analysis is based on a sociotechnical approach to management tools that considers organisational context, and the tool's technical substrate, theory of action and managerial philosophy.

FINDINGS

Results show that despite a standardised national mandate, the tool as implemented varied between organisations in terms of technical substrate and managerial philosophy. These variations are explained by the flexibility of the technical substrate, the lack of clarity of the managerial philosophy, and some contextual elements. Successful implementation may rest upon high hybridization of the tool on these different dimensions. A precise and prescribed technical substrate is not sufficient to guarantee implementation of a managerial philosophy.

PRACTICAL IMPLICATIONS

Mandated implementation of management tools may be more successful if it is explicit on the managerial philosophy, the technical substrate and the link between the two, and if it provides some leeway to adapt both to the organisational context.

ORIGINALITY/VALUE

This is one of the few studies to describe and analyse the process involved in mandated large-scale implementation of performance management systems in public healthcare organisations.

How to Work Collaboratively Within the Health System: Workshop Summary and Facilitator Reflection

Effectiveness in health services research requires development of specific knowledge and skills for working in partnership with health system decision-makers. In an initial effort to frame capacity-building activities for researchers, we designed a workshop on working collaboratively within the health system. The workshop, based on recent research exploring health system experience and perspectives on research collaborations, was trialed at the annual Canadian Health Services and Policy Research (CAHSPR) conference in May 2019. Participants reported positive evaluations of the workshop. However, further efforts should target health services researchers that may not be as motivated to develop skills in collaborative research. Additional attention to equipping researchers with the skills needed to work in partnerships is recommended, including approaches and materials that avoid oversimplification of complex challenges.

The unpredictable journeys of spreading, sustaining and scaling healthcare innovations: a scoping review

Innovation has the potential to improve the quality of care and health service delivery, but maximising the reach and impact of innovation to achieve large-scale health system transformation remains understudied. Interest is growing in three processes of the innovation journey within health systems, namely the spread, sustainability and scale-up (3S) of innovation. Recent reviews examine what we know about these processes. However, there is little research on how to support and operationalise the 3S. This study aims to improve our understanding of the 3S of healthcare innovations. We focus specifically on the definitions of the 3S, the mechanisms that underpin them, and the conditions that either enable or limit their potential. We conducted a scoping review, systematically investigating six bibliographic databases to search, screen and select relevant literature on the 3S of healthcare innovations. We screened 641 papers, then completed a full-text review of 112 identified as relevant based on title and abstract. A total of 24 papers were retained for analysis. Data were extracted and synthesised through descriptive and inductive thematic analysis. From this, we develop a framework of actionable guidance for health system actors aiming to leverage the 3S of innovation across five key areas of focus, as follows: (1) focus on the why, (2) focus on perceived-value and feasibility, (3) focus on what people do, rather than what they should be doing, (4) focus on creating a dialogue between policy and delivery, and (5) focus on inclusivity and capacity building. While there is no standardised approach to foster the 3S of healthcare innovations, a variety of practical frameworks and tools exist to support stakeholders along this journey.

Health system reforms in mature welfare states: tales from the north

Objective: This article has the objective show an essay on emerging themes in health system reforms, based on experience in Canada. Data synthesis: Reforms are the privileged mode of social change used by modern democratic societies. Persistent dysfunction and failure to adapt to emerging health needs and priorities within health systems in Canada provide a strong policy rationale to search for alternative strategies that might produce much-needed reforms. Three persistent challenges and opportunities for reform in Canadian health systems are discussed: the design of effective governance arrangements, the large-scale development and implementation of improvement and transformative capacities, and the leadership and engagement of the medical profession in working toward broad system goals. In exploring these challenges, we identify tensions that seem relevant to better understanding health system reform in mature welfare states. Conclusion: Addressing these tensions will require both a reinforcement of state and government capacities and stronger capacities at all levels of the health system to design and support change.