Encouraging translation and assessing impact of the Centre for Research Excellence in Integrated Quality Improvement: rationale and protocol for a research impact assessment

Reflecting on the quality of a methodologically pluralist evaluation of a large-scale Indigenous health research collaboration in Australia

BACKGROUND

Indigenous communities worldwide lead calls for all evaluations of research, programmes and policies affecting their communities to reflect the values, priorities and perspectives of the Indigenous peoples and communities involved. Tools, such as the Quality Appraisal Tool (QAT), are available to assess research quality through an Indigenous cultural lens. Good evaluation requires that evaluation efforts be evaluated. We found that critical reflection on the quality of evaluations from an Indigenous perspective is largely absent from the published literature. To ensure that we strive for quality in evaluation as determined by Indigenous people with whom we work, we examined the quality of our own evaluation of an Indigenous health research collaboration by conducting a reflexive dialogue.

METHODS

The QAT was used to assess our evaluation according to Indigenous health research principles. Our qualitative study used analytical coautoethnography to generate data through a series of reflexive dialogue sessions with Indigenous and non-Indigenous members of the research collaboration, using the QAT criteria as discussion prompts. Our ideas and reflections were compared and contrasted through a collaborative and iterative writing process, multiple review cycles and discussions.

RESULTS

We documented our findings against the QAT framework. We found examples that each QAT principle had, to some extent, been adhered to, but constantly needed to assess whether the principles were fully achieved to our satisfaction. Strengths of the evaluation included being adaptable and responsive to emerging issues for the research collaboration, while areas for improvement included more Indigenous leadership of, and involvement in, evaluation.

CONCLUSIONS

Although reflexive evaluation practice is not always comfortable, it does provide an opportunity to generate insights for improvement. Reflecting as we did-in a partnership between Indigenous and non-Indigenous colleagues-enabled deeper insights and meaning. We anticipate that our process models how other research in Indigenous contexts might better advance ethical, quality Indigenous research through working in collaboration with Indigenous researchers and communities.

What impact has the Centre of Research Excellence in Digestive Health made in the field of gastrointestinal health in Australia and internationally? Study protocol for impact evaluation using the FAIT framework

INTRODUCTION

The need for public research funding to be more accountable and demonstrate impact beyond typical academic outputs is increasing. This is particularly challenging and the science behind this form of research is in its infancy when applied to collaborative research funding such as that provided by the Australian National Health and Medical Research Council to the Centre for Research Excellence in Digestive Health (CRE-DH).

METHODS AND ANALYSIS

In this paper, we describe the protocol for applying the Framework to Assess the Impact from Translational health research to the CRE-DH. The study design involves a five-stage sequential mixed-method approach. In phase I, we developed an impact programme logic model to map the pathway to impact and establish key domains of benefit such as knowledge advancement, capacity building, clinical implementation, policy and legislation, community and economic impacts. In phase 2, we have identified and selected appropriate, measurable and timely impact indicators for each of these domains and established a data plan to capture the necessary data. Phase 3 will develop a model for cost-consequence analysis and identification of relevant data for microcosting and valuation of consequences. In phase 4, we will determine selected case studies to include in the narrative whereas phase 5 involves collation, data analysis and completion of the reporting of impact.We expect this impact evaluation to comprehensively describe the contribution of the CRE-DH for intentional activity over the CRE-DH lifespan and beyond to improve outcomes for people suffering with chronic and debilitating digestive disorders.

ETHICS AND DISSEMINATION

This impact evaluation study has been registered with the Hunter New England Human Research Ethics Committee as project 2024/PID00336 and ethics application 2024/ETH00290. Results of this study will be disseminated via medical conferences, peer-reviewed publications, policy submissions, direct communication with relevant stakeholders, media and social media channels such as X (formely Twitter).

Network evaluation of an innovation platform in continuous quality improvement in Australian Indigenous primary healthcare

Background

From 2014 to 2019, the Centre for Research Excellence in Integrated Quality Improvement (CRE-IQI) was evaluated as an innovation platform focusing on continuous quality improvement in Indigenous Australian primary healthcare. Although social network analysis (SNA) is a recognized method for evaluating the functioning, collaboration and effectiveness of innovation platforms, applied research is limited. This study applies SNA to evaluate the CRE-IQI’s functioning as an innovation platform.

Methods

Two surveys (2017, 2019) were conducted using social survey and network methods. Survey items covered respondent characteristics, their perceptions of the CRE-IQI’s performance, and its impact and sociometric relationships. Members’ relationship information was captured for the CRE-IQI at three time points, namely start (retrospectively), midpoint and final year, on three network types (knew, shared information, collaborated). SNA software was used to compute standard network metrics including diameter, density and centrality, and to develop visualizations. Survey and network results were addressed in a workshop held by members to develop improvement strategies.

Results

The response rate was 80% in 2017 and 65% in 2019 (n = 49 and 47, respectively). Between 2017 and 2019, respondents’ mean ratings of the CRE-IQI’s functioning and achievements in meeting its goals were sustained. They perceived the CRE-IQI as multidisciplinary, having effective management and governance, and incorporating Indigenous research leadership, representation and ways of working. Respondents recognized high levels of trust amongst members, rated “good communication and coordination with participants” highly, and “facilitating collaboration” as the CRE’s most strongly recognized achievement. In collaboration and information-sharing networks, average path length remained low in 2017 and 2019, indicating good small-world network properties for relaying information. On average, respondents shared information and collaborated with more CRE members in 2017 than 2019. However, in both 2017 and 2019 there were new collaborations and information-sharing outside of direct collaborations. CRE-IQI outcomes included: evidence generation; knowledge transfer and skills development in quality improvement; research capacity-building, career development; mentoring; grant support; development of new projects; health service support; and policy impact.

Conclusions

This study shows the utility of network analysis in evaluating the functioning, and collaboration, at the individual, organizational and health system levels, of an innovation platform, and adds to our understanding of factors enabling successful innovation platforms.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12961-022-00909-z.

Research protocol for impact assessment of a project to scale up food policies in the Pacific

Background

One of the challenges for countries implementing food policy measures has been the difficulty in demonstrating impact and retaining stakeholder support. Consequently, research funded to help countries overcome these challenges should assess impact and translation into practice, particularly in low-resource settings. However, there are still few attempts to prospectively, and comprehensively, assess research impact. This protocol describes a study co-created with project implementers, collaborative investigators and key stakeholders to optimize and monitor the impact of a research project on scaling up food policies in Fiji.

Methods

To develop this protocol, our team of researchers prospectively applied the Framework to Assess the Impact from Translational health research (FAIT). Activities included (i) developing a logic model to map the pathway to impact and establish domains of benefit; (ii) identifying process and impact indicators for each of these domains; (iii) identifying relevant data for impact indicators and a cost–consequence analysis; and (iv) establishing a process for collecting quantitative and qualitative data to measure progress. Impact assessment data will be collected between September 2022 and December 2024, through reports, routine monitoring activities, group discussions and semi-structured interviews with key implementers and stakeholders. The prospective application of the protocol, and interim and final research impact assessments of each project stream and the project as a whole, will optimize and enable robust measurement of research impact.

Discussion

By applying this protocol, we aim to increase understanding of pathways to impact and processes that need to be put in place to achieve this. This impact evaluation will inform future projects with a similar scope and will identify transferable and/or translatable lessons for other Pacific Island states and low- and middle-income countries.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12961-022-00927-x.

Methodological pluralism for better evaluations of complex interventions: lessons from evaluating an innovation platform in Australia

Complex interventions, such as innovation platforms, pose challenges for evaluators. A variety of methodological approaches are often required to build a more complete and comprehensive understanding of how complex interventions work. In this paper, we outline and critically appraise a methodologically pluralist evaluation of an innovation platform to strengthen primary care for Aboriginal and Torres Strait Islander Australians. In doing so, we aim to identify lessons learned from the approach taken and add to existing literature on implementing evaluations in complex settings, such as innovation platforms. The pluralist design used four evaluation approaches-developmental evaluation, principles-focused evaluation, network analysis, and framework analysis-with differing strengths and challenges. Taken together, the multiple evaluation approaches yielded a detailed description and nuanced understanding of the formation, functioning and outcomes of the innovation platform that would be difficult to achieve with any single evaluation method. While a methodologically pluralist design may place additional pressure on logistical and analytic resources available, it enables a deeper understanding of the mechanisms that underlie complex interventions.

Applying a framework to assess the impact of cardiovascular outcomes improvement research

Background

Health and medical research funding agencies are increasingly interested in measuring the impact of funded research. We present a research impact case study for the first four years of an Australian National Health and Medical Research Council funded Centre of Research Excellence in Cardiovascular Outcomes Improvement (2016–2020). The primary aim of this paper was to explore the application of a research impact matrix to assess the impact of cardiovascular outcomes improvement research.

Methods

We applied a research impact matrix developed from a systematic review of existing methodological frameworks used to measure research impact. This impact matrix was used as a bespoke tool to identify and understand various research impacts over different time frames. Data sources included a review of existing internal documentation from the research centre and publicly available information sources, informal iterative discussions with 10 centre investigators, and confirmation of information from centre grant and scholarship recipients.

Results

By July 2019, the impact on the short-term research domain category included over 41 direct publications, which were cited over 87 times (median journal impact factor of 2.84). There were over 61 conference presentations, seven PhD candidacies, five new academic collaborations, and six new database linkages conducted. The impact on the mid-term research domain category involved contributions towards the development of a national cardiac registry, cardiovascular guidelines, application for a Medicare Benefits Schedule reimbursement item number, introduction of patient-reported outcome measures into several databases, and the establishment of nine new industry collaborations. Evidence of long-term impacts were described as the development and use of contemporary management for aortic stenosis, a cardiovascular risk prediction model and prevention targets in several data registries, and the establishment of cost-effectiveness for stenting compared to surgery.

Conclusions

We considered the research impact matrix a feasible tool to identify evidence of academic and policy impact in the short- to midterm; however, we experienced challenges in capturing long-term impacts. Cost containment and broader economic impacts represented another difficult area of impact to measure.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12961-021-00710-4.

What was the impact of a participatory research project in Australian Indigenous primary healthcare services? Applying a comprehensive framework for assessing translational health research to Lessons for the Best

OBJECTIVES

To (1) apply the Framework to Assess the Impact from Translational health research (FAIT) to Lessons from the Best to Better the Rest (LFTB), (2) report on impacts from LFTB and (3) assess the feasibility and outcomes from a retrospective application of FAIT.

SETTING

Three Indigenous primary healthcare (PHC) centres in the Northern Territory, Australia; project coordinating centre distributed between Townsville, Darwin and Cairns and the broader LFTB learning community across Australia.

PARTICIPANTS

LFTB research team and one representative from each PHC centre.

PRIMARY AND SECONDARY OUTCOME MEASURES

Impact reported as (1) quantitative metrics within domains of benefit using a modified Payback Framework, (2) a cost-consequence analysis given a return on investment was not appropriate and (3) a narrative incorporating qualitative evidence of impact. Data were gathered through in-depth stakeholder interviews and a review of project documentation, outputs and relevant websites.

RESULTS

LFTB contributed to knowledge advancement in Indigenous PHC service delivery; enhanced existing capacity of health centre staff, researchers and health service users; enhanced supportive networks for quality improvement; and used a strengths-based approach highly valued by health centres. LFTB also leveraged between $A1.4 and $A1.6 million for the subsequent Leveraging Effective Ambulatory Practice (LEAP) Project to apply LFTB learnings to resource development and creation of a learning community to empower striving PHC centres.

CONCLUSION

Retrospective application of FAIT to LFTB, although not ideal, was feasible. Prospective application would have allowed Indigenous community perspectives to be included. Greater appreciation of the full benefit of LFTB including a measure of return on investment will be possible when LEAP is complete. Future assessments of impact need to account for the limitations of fully capturing impact when intermediate/final impacts have not yet been realised and captured.

Using developmental evaluation to enhance continuous reflection, learning and adaptation of an innovation platform in Australian Indigenous primary healthcare

Effective efforts to strengthen health systems need diverse, multi-stakeholder networks working together on complex or 'wicked' problems such as prevention and control of chronic diseases, solutions to which go beyond the role and capability of one organisation. The contextual complexities inherent in 'wicked' problems mean that solutions warrant a systems approach that encompasses innovation and new ways of thinking about, facilitating and implementing collective decision-making processes and change practices.Innovation platforms are a mechanism for facilitating communication and collaboration among diverse stakeholders, promoting joint action and stimulating innovation. Developmental evaluation is an approach that is increasingly being used to evaluate innovative and emergent programmes and projects, as it enables evaluators to provide real-time feedback so that evaluation findings can be used to guide development and adaptations. Developmental evaluation emphasises learning and adaptation, and aligns well with the implementation of innovation platforms that have continuous reflection, learning and adaptation as a specific design principle.Here, we outline our rationale for applying a developmental evaluation to enhance the formation, functioning and outcomes of an innovation platform aimed at accelerating and strengthening large-scale quality improvement efforts in Australian Aboriginal and Torres Strait Islander primary healthcare. We provide examples to explain how the developmental evaluation findings were used for adaptation of the innovation platform and assess to what extent our application of developmental evaluation was consistent with, and reflective of, its essential principles.Our evaluation aligned strongly with the principles of developmental evaluation, and the approach we took was well suited to situations with a developmental purpose, innovation niche and complexity such as innovation platforms. As a result, along with the increasing interest in multi-stakeholder platforms (e.g. innovation platforms) and the inherent challenges with evaluating these complex networks, we anticipate our use of this approach being of interest globally.

Strengthening and measuring research impact in global health: lessons from applying the FAIT framework

Background

To date, efforts to measure impact have largely focused on health research in high-income countries, reflecting where the majority of health research funding is spent. Nevertheless, there is a growing body of health and medical research being undertaken in low- and middle-income countries (LMICs), supported by both development aid and established research funders. The Framework to Assess the Impact of Translational health research (FAIT) combines three approaches to measuring research impact (Payback, economic assessment and case study narrative). Its aim is to strengthen the focus on translation and impact measurement in health research. FAIT has been used by several Australian research initiatives; however, it has not been used in LMICs. Our aim was to apply FAIT in an LMIC context and evaluate its utility.

Methods

We retrospectively applied all three FAIT methods to two LMIC studies using available data, supplemented with group discussion and further economic analyses. Results were presented in a scorecard format.

Results

FAIT helped clarify pathways of impact for the projects and provided new knowledge on areas of impact in several domains, including capacity-building for research, policy development and economic impact. However, there were constraints, particularly associated with calculating the return on investment in the LMIC context. The case study narrative provided a layperson’s summary of the research that helped to explain outcomes and succinctly communicate lessons learnt.

Conclusion

Use of FAIT to assess the impact of LMIC research was both feasible and useful. We make recommendations related to prospective use, identification of metrics to support use of the Payback framework, and simplification of the economic assessment, which may facilitate further application in LMIC environments.

Electronic supplementary material

The online version of this article (10.1186/s12961-019-0451-0) contains supplementary material, which is available to authorized users.

Comparing and contrasting 'innovation platforms' with other forms of professional networks for strengthening primary healthcare systems for Indigenous Australians

Efforts to strengthen health systems require the engagement of diverse, multidisciplinary stakeholder networks. Networks provide a forum for experimentation and knowledge creation, information exchange and the spread of good ideas and practice. They might be useful in addressing complex issues or ‘wicked’ problems, the solutions to which go beyond the control and scope of any one agency.

Innovation platforms are proposed as a novel type of network because of their diverse stakeholder composition and focus on problem solving within complex systems. Thus, they have potential applicability to health systems strengthening initiatives, even though they have been predominantly applied in the international agricultural development sector.

In this paper, we compare and contrast the concept of innovation platforms with other types of networks that can be used in efforts to strengthen primary healthcare systems, such as communities of practice, practice-based research networks and quality improvement collaboratives. We reflect on our ongoing research programme that applies innovation platform concepts to drive large-scale quality improvement in primary healthcare for Aboriginal and Torres Strait Islander Australians and outline our plans for evaluation. Lessons from our experience will find resonance with others working on similar initiatives in global health.

An "All Teach, All Learn" Approach to Research Capacity Strengthening in Indigenous Primary Health Care Continuous Quality Improvement

In Australia, Indigenous people experience poor access to health care and the highest rates of morbidity and mortality of any population group. Despite modest improvements in recent years, concerns remains that Indigenous people have been over-researched without corresponding health improvements. Embedding Indigenous leadership, participation, and priorities in health research is an essential strategy for meaningful change for Indigenous people. To centralize Indigenous perspectives in research processes, a transformative shift away from traditional approaches that have benefited researchers and non-Indigenous agendas is required. This shift must involve concomitant strengthening of the research capacity of Indigenous and non-Indigenous researchers and research translators-all must teach and all must learn. However, there is limited evidence about how to strengthen systems and stakeholder capacity to participate in and lead continuous quality improvement (CQI) research in Indigenous primary health care, to the benefit of Indigenous people. This paper describes the collaborative development of, and principles underpinning, a research capacity strengthening (RCS) model in a national Indigenous primary health care CQI research network. The development process identified the need to address power imbalances, cultural contexts, relationships, systems requirements and existing knowledge, skills, and experience of all parties. Taking a strengths-based perspective, we harnessed existing knowledge, skills and experiences; hence our emphasis on capacity "strengthening". New insights are provided into the complex processes of RCS within the context of CQI in Indigenous primary health care.