A developmental evaluation to enhance stakeholder engagement in a wide-scale interactive project disseminating quality improvement data: study protocol for a mixed-methods study

Developing a compassionate university: Insights from a longitudinal process evaluation

Compassionate communities are gaining momentum as a new public health approach emphasizing community support during times of serious illness, death, and bereavement. However, evidence on their development, particularly in higher education, is limited. This study investigates the development of a Compassionate University, examining the underlying processes and contextual factors shaping its development. A longitudinal process evaluation was conducted, using field notes right-now surveys, individual interviews, focus groups, and strategic learning debriefs. Factors that facilitated the development process included leadership support, the establishment of the Compassionate Schools Learning Network, and alignment with existing university programs. Barriers were the lack of guiding examples, the fragmented university environment, resource constraints, and limited prioritization. Cognitive and social processes that supported the work involved recognizing the value of Compassionate University and adapting implementation strategies based on empirical feedback. However, challenges such as building coherence, engaging stakeholders, and assessing the work hampered the development process.

Stakeholder theory and management: Understanding longitudinal collaboration networks

This paper explores the evolution of research collaboration networks in the ‘stakeholder theory and management’ (STM) discipline and identifies the longitudinal effect of co-authorship networks on research performance, i.e., research productivity and citation counts. Research articles totaling 6,127 records from 1989 to 2020 were harvested from the Web of Science Database and transformed into bibliometric data using Bibexcel, followed by applying social network analysis to compare and analyze scientific collaboration networks at the author, institution and country levels. This work maps the structure of these networks across three consecutive sub-periods (t₁: 1989–1999; t₂: 2000–2010; t₃: 2011–2020) and explores the association between authors’ social network properties and their research performance. The results show that authors collaboration network was fragmented all through the periods, however, with an increase in the number and size of cliques. Similar results were observed in the institutional collaboration network but with less fragmentation between institutions reflected by the increase in network density as time passed. The international collaboration had evolved from an uncondensed, fragmented and highly centralized network, to a highly dense and less fragmented network in t₃. Moreover, a positive association was reported between authors’ research performance and centrality and structural hole measures in t₃ as opposed to ego-density, constraint and tie strength in t₁. The findings can be used by policy makers to improve collaboration and develop research programs that can enhance several scientific fields. Central authors identified in the networks are better positioned to receive government funding, maximize research outputs and improve research community reputation. Viewed from a network’s perspective, scientists can understand how collaborative relationships influence research performance and consider where to invest their decision and choices.

Appreciative inquiry in evaluating integrated primary oral health services in Quebec Cree communities: a qualitative multiple case study

OBJECTIVE

The Strategic Regional Plan of the Cree Board of Health and Social Services of James Bay (CBHSSJB), serving the Quebec Cree communities, mandates the objective of integrating oral health within primary healthcare. Emerging evidence suggests that the integration of oral health into primary healthcare can decrease oral health disparities. This research study aimed to answer the following research question: how and to what extent does the integration of oral health into primary healthcare address the oral health needs of the Cree communities?

DESIGN

We used a multiple-case study design within a qualitative approach and developmental evaluation methodology. The Discovery, Dream, Design and Destiny model of appreciative inquiry was selected as a study framework among existing frameworks of the developmental evaluation.

SETTING

Four purposefully selected Cree communities.

PARTICIPANTS

Healthcare providers, administrators and patients at the community wellness centres and hospital.

OUTCOME MEASURES

Integration of oral health into primary healthcare.

RESULTS

A total of 36 interviews and 6 focus group discussions were conducted. We identified ten themes in discovery and dream phases. The Discovery phase identified the strengths of the organisation in facilitating enablers of integration including strategic planning, organisational structure, cultural integration, coordinated networks and colocation. In the Dream phase, participants' oral healthcare stories expressed various dimensions of integration and their wish for strengthening integration via extending public oral healthcare programmes, increasing resources and improving organisational management. In the Design phase, recommendations were formulated for a future action plan within the CBHSSJB.

CONCLUSION

This study results suggested that the CBHSSJB has been successful in implementing oral health integration into primary care following its strategic planning. At present, the organisation could extend the level of integration into full integration by following study recommendations derived from the perspective of local stakeholders.

Using developmental evaluation to enhance continuous reflection, learning and adaptation of an innovation platform in Australian Indigenous primary healthcare

Effective efforts to strengthen health systems need diverse, multi-stakeholder networks working together on complex or 'wicked' problems such as prevention and control of chronic diseases, solutions to which go beyond the role and capability of one organisation. The contextual complexities inherent in 'wicked' problems mean that solutions warrant a systems approach that encompasses innovation and new ways of thinking about, facilitating and implementing collective decision-making processes and change practices.Innovation platforms are a mechanism for facilitating communication and collaboration among diverse stakeholders, promoting joint action and stimulating innovation. Developmental evaluation is an approach that is increasingly being used to evaluate innovative and emergent programmes and projects, as it enables evaluators to provide real-time feedback so that evaluation findings can be used to guide development and adaptations. Developmental evaluation emphasises learning and adaptation, and aligns well with the implementation of innovation platforms that have continuous reflection, learning and adaptation as a specific design principle.Here, we outline our rationale for applying a developmental evaluation to enhance the formation, functioning and outcomes of an innovation platform aimed at accelerating and strengthening large-scale quality improvement efforts in Australian Aboriginal and Torres Strait Islander primary healthcare. We provide examples to explain how the developmental evaluation findings were used for adaptation of the innovation platform and assess to what extent our application of developmental evaluation was consistent with, and reflective of, its essential principles.Our evaluation aligned strongly with the principles of developmental evaluation, and the approach we took was well suited to situations with a developmental purpose, innovation niche and complexity such as innovation platforms. As a result, along with the increasing interest in multi-stakeholder platforms (e.g. innovation platforms) and the inherent challenges with evaluating these complex networks, we anticipate our use of this approach being of interest globally.

Using developmental evaluation to support knowledge translation: reflections from a large-scale quality improvement project in Indigenous primary healthcare

BACKGROUND

Developmental evaluation is a growing area of evaluation practice, advocated for informing the adaptive development of change initiatives in complex social environments. The utilisation focus, complexity perspective and systems thinking of developmental evaluation suggest suitability for evaluating knowledge translation initiatives in primary healthcare. However, there are few examples in the literature to guide its use in these contexts and in Indigenous settings. In this paper, we reflect on our experience of using developmental evaluation to implement a large-scale knowledge translation research project in Australian Aboriginal and Torres Strait Islander primary healthcare. Drawing on principles of knowledge translation and key features of developmental evaluation, we debate the key benefits and challenges of applying this approach to engage diverse stakeholders in using aggregated quality improvement data to identify and address persistent gaps in care delivery.

DISCUSSION

The developmental evaluation enabled the team to respond to stakeholder feedback and apply learning in real-time to successfully refine theory-informed research and engagement processes, tailor the presentation of findings to stakeholders and context, and support the project's dissemination and knowledge co-production aim. It thereby contributed to the production of robust, useable research findings for informing policy and system change. The use of developmental evaluation appeared to positively influence stakeholders' use of the project reports and their responses to the findings. Challenges included managing a high volume of evaluation data and multiple evaluation purposes, balancing facilitative sense-making processes and change with task-focused project management, and lack of experience in using this evaluation approach. Use of an embedded evaluator with facilitation skills and background knowledge of the project helped to overcome these challenges, as did similarities observed between features of developmental evaluation and continuous quality improvement.

CONCLUSION

Our experience of developmental evaluation confirmed our expectations of the potential value of this approach for strengthening improvement interventions and implementation research, and particularly for adapting healthcare innovations in Indigenous settings. In our project, developmental evaluation successfully encompassed evaluation, project adaptation, capacity development and knowledge translation. Further work is warranted to apply this approach more widely to improve primary healthcare initiatives and outcomes, and to evaluate implementation research.

Wide-Scale Continuous Quality Improvement: A Study of Stakeholders' Use of Quality of Care Reports at Various System Levels, and Factors Mediating Use

Introduction: Increasing the use of evidence in healthcare policy and practice requires greater understanding of how stakeholders use evidence to inform policy, refine systems and change practice. Drawing on implementation theory, we have used system-focused participatory research to engage diverse stakeholders in using aggregated continuous quality improvement (CQI) data from Australian Indigenous primary health care settings to identify priority evidence-practice gaps, barriers/enablers and strategies for improvement. This article reports stakeholders' use or intended use of evidence at various levels of the system, and factors mediating use. Material and Methods: Interviews were undertaken with a purposeful sample of 30 healthcare stakeholders in different roles, organization types and settings in one Australian jurisdiction and with national participants, as part of the project's developmental evaluation. Qualitative data were analyzed to identify themes and categories relating to use of evidence. Results: Context-specific aggregated CQI data that were relatable to the diverse professional roles and practices provided an effective starting point for sharing perspectives, generating practice-based evidence and mobilizing evidence-use. Interviewees perceived the co-produced findings as applicable at different levels and useful for planning, policy development, supporting best practice and reflection, capacity strengthening and developing new research. Factors mediating use were commitment to best practice; the credibility of the evidence and its perceived relevance to work roles, contexts and decision needs; report format and language; facilitation and communication; competing work pressures and the organizational environment for change. Conclusions: This study found that primary health care stakeholders used evidence on quality of care for a variety of purposes. This could be linked to the interactive research processes used to engage stakeholders in different roles and settings in interpreting data, sharing and generating knowledge. Findings indicate that system-based participatory research using CQI data and iterative, interactive and systematic CQI-based methods can be applied at scale to support concurrent action for healthcare improvement at different system levels. Factors known to influence implementation should be addressed within the research design to optimize evidence use. Further research is needed to explore the utility of interactive dissemination for engaging healthcare stakeholders in informing policy and system change.

Application of the i-PARIHS framework for enhancing understanding of interactive dissemination to achieve wide-scale improvement in Indigenous primary healthcare

Background

Participatory research approaches improve the use of evidence in policy, programmes and practice. Few studies have addressed ways to scale up participatory research for wider system improvement or the intensity of effort required. We used the integrated Promoting Action on Research Implementation in Health Services (i-PARIHS) framework to analyse implementation of an interactive dissemination process engaging stakeholders with continuous quality improvement (CQI) data from Australian Indigenous primary healthcare centres. This paper reports lessons learnt about scaling knowledge translation research, facilitating engagement at a system level and applying the i-PARIHS framework to a system-level intervention.

Methods

Drawing on a developmental evaluation of our dissemination process, we conducted a post-hoc analysis of data from project records and interviews with 30 stakeholders working in Indigenous health in different roles, organisation types and settings in one Australian jurisdiction and with national participants. Content-analysed data were mapped onto the i-PARIHS framework constructs to examine factors contributing to the success (or otherwise) of the process.

Results

The dissemination process achieved wide reach, with stakeholders using aggregated CQI data to identify system-wide priority evidence–practice gaps, barriers and strategies for improvement across the scope of care. Innovation characteristics influencing success were credible data, online dissemination and recruitment through established networks, research goals aligned with stakeholders’ interest in knowledge-sharing and motivation to improve care, and iterative phases of reporting and feedback. The policy environment and infrastructure for CQI, as well as manager support, influenced participation. Stakeholders who actively facilitated organisational- and local-level engagement were important for connecting others with the data and with the externally located research team. Developmental evaluation was facilitative in that it supported real-time adaptation and tailoring to stakeholders and context.

Conclusions

A participatory research process was successfully implemented at scale without intense facilitation efforts. These findings broaden the notion of facilitation and support the utility of the i-PARIHS framework for planning participatory knowledge translation research at a system level. Researchers planning similar interventions should work through established networks and identify organisational- or local-level facilitators within the research design. Further research exploring facilitation in system-level interventions and the use of interactive dissemination processes in other settings is needed.

Electronic supplementary material

The online version of this article (10.1186/s12961-018-0392-z) contains supplementary material, which is available to authorized users.

Comparing and contrasting 'innovation platforms' with other forms of professional networks for strengthening primary healthcare systems for Indigenous Australians

Efforts to strengthen health systems require the engagement of diverse, multidisciplinary stakeholder networks. Networks provide a forum for experimentation and knowledge creation, information exchange and the spread of good ideas and practice. They might be useful in addressing complex issues or ‘wicked’ problems, the solutions to which go beyond the control and scope of any one agency.

Innovation platforms are proposed as a novel type of network because of their diverse stakeholder composition and focus on problem solving within complex systems. Thus, they have potential applicability to health systems strengthening initiatives, even though they have been predominantly applied in the international agricultural development sector.

In this paper, we compare and contrast the concept of innovation platforms with other types of networks that can be used in efforts to strengthen primary healthcare systems, such as communities of practice, practice-based research networks and quality improvement collaboratives. We reflect on our ongoing research programme that applies innovation platform concepts to drive large-scale quality improvement in primary healthcare for Aboriginal and Torres Strait Islander Australians and outline our plans for evaluation. Lessons from our experience will find resonance with others working on similar initiatives in global health.