Assessing the impact of healthcare research: A systematic review of methodological frameworks

A multidimensional assessment of payback: Evaluating the impact of Breast Cancer Trials

AIM

This study evaluates the impact created by clinical trial investment and research undertaken by Breast Cancer Trials (BCT).

METHODS

The approach involved using the Payback Framework, and included a review of BCT archival information, public health data, a questionnaire sent to BCT members, individual interviews with key informants, a focus group with members of the organization's Consumer Advisory Panel, and case studies of select BCT trials. The evaluation assessed the evidence against the Payback Framework criteria: informing policy and product development, knowledge production, the research system, health and health sector benefits, and broader economic benefits.

RESULTS

Analysis using the Payback Framework revealed impact was created in each category and a range of positive outcomes were identified for various stakeholder groups. BCT is maximizing the impact of its research and contributing to a global pool of scientific knowledge by collaborating with over 100 institutions and 820 researchers, yet its benefits go beyond research contributions. Findings highlight strong financial returns from BCT's research program and that BCT has played an important role in enhancing the public perception of clinical trials by promoting participation in trials, educating and empowering participants, and communicating trial outcomes.

CONCLUSION

BCT's clinical trials have had a significant impact on the prevention, detection, treatment, and management of breast cancer. Measuring and reporting impact can be resource intensive but support BCT in remaining accountable to all those invested in the organization and in breast cancer clinical trials, evidencing the multiple dimensions of payback resulting from the organization's research.

The Conceptualization and Measurement of Research Impact in Primary Health Care: Protocol for a Rapid Scoping Review

BACKGROUND

The generation of research evidence and knowledge in primary health care (PHC) is crucial for informing the development and implementation of interventions and innovations and driving health policy, health service improvements, and potential societal changes. PHC research has broad effects on patients, practices, services, population health, community, and policy formulation. The in-depth exploration of the definition and measures of research impact within PHC is essential for broadening our understanding of research impact in the discipline and how it compares to other health services research.

OBJECTIVE

The objectives of the study are (1) to understand the conceptualizations and measures of research impact within the realm of PHC and (2) to identify methodological frameworks for evaluation and research impact and the benefits and challenges of using these approaches. The forthcoming review seeks to guide future research endeavors and enhance methodologies used in assessing research impact within PHC.

METHODS

The protocol outlines the rapid review and environmental scan approach that will be used to explore research impact in PHC and will be guided by established frameworks such as the Canadian Academy of Health Sciences Impact Framework and the Canadian Health Services and Policy Research Alliance. The rapid review follows scoping review guidelines (PRISMA-ScR; Preferred Reporting Items for Systematic Review and Meta-Analysis Extension for Scoping Reviews). The environmental scan will be done by consulting with professional organizations, academic institutions, information science, and PHC experts. The search strategy will involve multiple databases, citation and forward citation searching, and manual searches of gray literature databases, think tank websites, and relevant catalogs. We will include gray and scientific literature focusing explicitly on research impact in PHC from high-income countries using the World Bank classification. Publications published in English from 1978 will be considered. The collected papers will undergo a 2-stage independent review process based on predetermined inclusion criteria. The research team will extract data from selected studies based on the research questions and the CRISP (Consensus Reporting Items for Studies in Primary Care) protocol statement. The team will discuss the extracted data, enabling the identification and categorization of key themes regarding research impact conceptualization and measurement in PHC. The narrative synthesis will evolve iteratively based on the identified literature.

RESULTS

The results of this study are expected at the end of 2024.

CONCLUSIONS

The forthcoming review will explore the conceptualization and measurement of research impact in PHC. The synthesis will offer crucial insights that will guide subsequent research, emphasizing the need for a standardized approach that incorporates diverse perspectives to comprehensively gauge the true impact of PHC research. Furthermore, trends and gaps in current methodologies will set the stage for future studies aimed at enhancing our understanding and measurement of research impact in PHC.

INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID)

PRR1-10.2196/55860.

Consequences of the hospital nursing research culture: Perspective of staff nurses

Objective

Research plays a critical role in molding nursing as a profession. Healthcare organizations are challenged to build an organizational culture that cultivates the development of nursing research. Creating a culture of nursing research is constantly branded a vital component in advancing nursing science. Therefore, this study described nurses' perceptions about their hospital's nursing research culture.

Methods

Semi-structured personal interviews were used to collect data. Seventeen staff nurses were selected to participate in the study by purposive sampling method from December 2022 to April 2023 in the eastern Philippines. The researchers used Braun and Clarke's thematic approach to analyze the qualitative data.

Results

The analysis revealed three central themes and seven subthemes: influencing behavior and performance of the employee (help in research information and learning, flexibility and adaptability, collaboration and communication among employees), influencing staff nurses' research participation (leadership strategies, training and professional development), and affecting hospitals to adapt to change (organizational performance and success, employee engagement and job satisfaction).

Conclusions

The findings support the benefits of a positive nursing research culture in hospitals due to its critical role in shaping nurses' behavior and performance, participation in nursing research, and hospital adaptation to changes. Collectively, these themes contribute to a deeper understanding of the multifaceted impact that a culture of nursing research can have on various levels of the healthcare system.

A toolkit for capturing a representative and equitable sample in health research

Research participants often do not represent the general population. Systematic exclusion of particular groups from research limits the generalizability of research findings and perpetuates health inequalities. Groups considered underserved by research include those whose inclusion is lower than expected based on population estimates, those with a high healthcare burden but limited research participation opportunities and those whose healthcare engagement is less than others. The REP-EQUITY toolkit guides representative and equitable inclusion in research. The toolkit was developed through a methodological systematic review and synthesis and finalized in a consensus workshop with 24 participants. The REP-EQUITY toolkit describes seven steps for investigators to consider in facilitating representative and equitable sample selection. This includes clearly defining (1) the relevant underserved groups, (2) the aims relating to equity and representativeness, (3) the sample proportion of individuals with characteristics associated with being underserved by research, (4) the recruitment goals, (5) the strategies by which external factors will be managed, (6) the methods by which representation in the final sample will be evaluated and (7) the legacy of having used the toolkit. Using the REP-EQUITY toolkit could promote trust between communities and research institutions, increase diverse participation in research and improve the generalizability of health research. National Institute for Health and Care Research PROSPERO identifier: CRD42022355391.

Investigating the research capacity and productivity of Canadian sports chiropractors

Objectives

To investigate the research capacity and productivity of Canadian sports chiropractors.

Methods

A cross-sectional survey (phase one) and scoping review (phase two) investigated the research capacity and productivity (from 2015-2020) of the Canadian sports chiropractic field.

Results

Most respondents (72%) reported obtaining research training from fellowship and master's programs, with only 2 (1%) PhD qualifications identified. Approximately, 30% reported active involvement in research, with 28% being part-time clinician researchers. Access to human and technological research resources were limited. We identified 67 publications and 16 conference presentations within a five-year period, with clinical, population health, and basic science research as the areas most studied.

Conclusion

The research effort of Canadian sports chiropractors is primarily conducted by clinicians involved in research on a part-time basis. Its research outputs predominantly reflect the research requirements of the RCCSS(C) Sports Sciences Residency Program, highlighting its contribution in developing capacity and producing research for the Canadian sports chiropractic field.

Development of a novel and more holistic approach for assessing impact in health and medical research: the Research Impact Assessment Framework

Considered investment in health and medical research (HMR) is critical for fostering a healthcare system that is sustainable, effective, responsive, and innovative. While several tools exist to measure the impact of research, few assess the research environment that nurtures and supports impactful research and the strategic alignment of research with societal needs. This perspective article discusses the limitations of existing assessment tools and presents a novel Research Impact Assessment Framework designed to enable more strategic and targeted investment towards HMR, having the potential for significant public benefit.

Using evidence in mental health policy agenda-setting in low- and middle-income countries: a conceptual meta-framework from a scoping umbrella review

The purpose of this article is to close the gap in frameworks for the use of evidence in the mental health policy agenda-setting in low- and middle-income countries (LMICs). Agenda-setting is important because mental health remains a culturally sensitive and neglected issue in LMICs. Moreover, effective evidence-informed agenda-setting can help achieve, and sustain, the status of mental health as a policy priority in these low-resource contexts. A scoping 'review of reviews' of evidence-to-policy frameworks was conducted, which followed preferred reporting items for systematic reviews and meta-analyses (PRISMA) guidelines. Nineteen reviews met the inclusion criteria. A meta-framework was developed from analysis and narrative synthesis of these 19 reviews, which integrates the key elements identified across studies. It comprises the concepts of evidence, actors, process, context and approach, which are linked via the cross-cutting dimensions of beliefs, values and interests; capacity; power and politics; and trust and relationships. Five accompanying questions act as a guide for applying the meta-framework with relevance to mental health agenda-setting in LMICs. This is a novel and integrative meta-framework for mental health policy agenda-setting in LMICs and, as such, an important contribution to this under-researched area. Two major recommendations are identified from the development of the framework to enhance its implementation. First, given the paucity of formal evidence on mental health in LMICs, informal evidence based on stakeholder experience could be better utilized in these contexts. Second, the use of evidence in mental health agenda-setting in LMICs would be enhanced by involving a broader range of stakeholders in generating, communicating and promoting relevant information.

Considerations for patient and public involvement and engagement in health research

Patient and public involvement and engagement (PPIE) can provide valuable insights into the experiences of those living with and affected by a disease or health condition. Inclusive collaboration between patients, the public and researchers can lead to productive relationships, ensuring that health research addresses patient needs. Guidelines are available to support effective PPIE; however, evaluation of the impact of PPIE strategies in health research is limited. In this Review, we evaluate the impact of PPIE in the 'Therapies for Long COVID in non-hospitalised individuals' (TLC) Study, using a combination of group discussions and interviews with patient partners and researchers. We identify areas of good practice and reflect on areas for improvement. Using these insights and the results of a survey, we synthesize two checklists of considerations for PPIE, and we propose that research teams use these checklists to optimize the impact of PPIE for both patients and researchers in future studies.

Development of a framework and research impact capture tool for nursing, midwifery, allied health professions, healthcare science, pharmacy and psychology (NMAHPPs)

BACKGROUND

There is an ambitious target to create a UK clinical academic workforce representing 1% of clinicians from nursing, midwifery, the allied health professions, healthcare science, pharmacy and psychology (NMAHPPs). Understanding and recording the impact that clinical academics make across healthcare services is crucial if we are to grow, value and support this highly skilled workforce group. However, it is currently difficult to systematically record, collate and report the impacts associated with NMAHPP research activity. The aims of this project were to i) develop a framework outlining the impacts that were important for key stakeholder groups, and ii) create and pilot a research impact capture tool to record these impacts.

METHODS

The framework was developed from the existing literature. It was refined, remodelled and approved by multidisciplinary stakeholder involvement, including patient and public representatives, healthcare managers and research-active clinicians. The framework was converted into a series of questions to create an electronic research impact capture tool, which was also refined through feedback from these stakeholder groups. The impact capture tool was piloted with research-active clinicians across a large NHS Trust and its associated organisations.

RESULTS

The impact framework contained eight elements: clinical background, research and service improvement activities, research capacity building, research into practice, patients and service users, research dissemination, economics and research funding, and collaborations. Thirty individuals provided data for the research impact capture tool pilot (55% response rate). Respondents reported a range of positive impacts representing all elements of the framework. Importantly, research-activity appeared to be a key driver for recruitment and retention in the sample population.

CONCLUSIONS

The impact capture tool is a feasible method of recording the breadth of impacts associated with NMAHPP research activity. We encourage other organisations to collaboratively use and refine our impact capture tool, with the aim of standardising reporting, and facilitating discussions about research activity within clinical appraisal. Pooling and comparing data will also allow comparison between organisations, and assessment of change over time or after implementation of interventions aimed at supporting and increasing research activity.

The impact of participation in research for speech and language therapy departments and their patients: A case example of the Big CACTUS multicentre trial of self-managed computerized aphasia therapy

BACKGROUND

In order to conduct research that is meaningful to speech and language therapy services and their patients, it is often desirable to conduct the research within routine clinical services. This can require considerable time and commitment from speech and language therapists (SLTs). It is therefore important to understand the impact that such participation in research can have.

AIMS

To explore the impact of research participation in the Big CACTUS study of self-managed computerized aphasia therapy conducted in 21 UK NHS speech and language therapy departments.

METHODS & PROCEDURES

An online survey was sent to SLTs who took the lead role for the study at their NHS Trust to evaluate the impact of study participation in three domains: capacity-building, research development and health services. The questionnaire, based on the VICTOR framework for evaluating research impact, included Likert scale statements and closed and open-ended questions. The results from open-ended questions were coded and analysed using framework analysis in NVivo 12 and the data from closed questions were analysed descriptively.

OUTCOMES & RESULTS

A total of 12 SLTs returned the survey. Nine codes were identified from open-ended questions and 20 predefined from the literature. Analysis of the responses demonstrated the perceived impact including improvements in practices and access to therapy, investments in infrastructure, increased SLT profile, and impact on research culture among SLTs. The usefulness of the intervention during the COVID-19 pandemic was also highlighted.

CONCLUSIONS & IMPLICATIONS

The results suggest participation in Big CACTUS has resulted in improvements in patient care and SLT research capacity and culture in speech and language therapy departments.

WHAT THIS PAPER ADDS

What is already known on the subject Practice-based research is encouraged to assist with the clinical relevance of the research findings. Participation in research can be seen as an activity that is additional to the core business of patient care and it can be difficult to secure time to participate or conduct research in clinical settings. Impact evaluation initiatives of individual trials facilitate early identification of benefits beyond the trial. What this paper adds to existing knowledge This study describes specific examples of the impact on services, staff and patients from SLT participation and leadership in the Big CACTUS speech and language therapy trial in clinical settings. What are the potential or actual clinical implications of this study? Clinical services participating in research may benefit from improved clinical care for patients both during and after the study, an improved professional reputation, and increased research capacity and culture within the clinical settings.

Impact assessment of the Centre for Research Excellence in Stroke Rehabilitation and Brain Recovery

BACKGROUND

Research impact is an emerging measure of research achievement alongside traditional academic outputs such as publications. We present the results of applying the Framework to Assess the Impact from Translational health research (FAIT) to the Centre for Research Excellence (CRE) in Stroke Rehabilitation and Brain Recovery (CRE-Stroke, 2014-2019) and report on the feasibility and lessons from the application of FAIT to a CRE rather than a discrete research project.

METHODS

Data were gathered via online surveys, in-depth interviews, document analysis and review of relevant websites/databases to report on the three major FAIT methods: the modified Payback Framework, an assessment of costs against monetized consequences, and a narrative account of the impact generated from CRE-Stroke activities. FAIT was applied during the last 4 years of CRE-Stroke operation.

RESULTS

With an economic investment of AU$ 3.9 million over 5 years, CRE-Stroke delivered a return on investment that included AU$ 18.8 million in leveraged grants, fellowships and consultancies. Collectively, CRE-Stroke members produced 354 publications that were accessed 470,000 times and cited over 7220 times. CRE-Stroke supported 26 PhDs, 39 postdocs and seven novice clinician researchers. There were 59 capacity-building events benefiting 744 individuals including policy-makers and consumers. CRE-Stroke created research infrastructure (including a research register of stroke survivors and a brain biobank), and its global leadership produced international consensus recommendations to influence the stroke research landscape worldwide. Members contributed to the Australian Living Stroke Guidelines: four researchers' outputs were directly referenced. Based only on the consequences that could be monetized, CRE-Stroke returned AU$ 4.82 for every dollar invested in the CRE.

CONCLUSION

This case example in the developing field of impact assessment illustrates how researchers can use evidence to demonstrate and report the impact of and returns on research investment. The prospective application of FAIT by a dedicated research impact team demonstrated impact in broad categories of knowledge-gain, capacity-building, new infrastructure, input to policy and economic benefits. The methods can be used by other research teams to provide comprehensive evidence to governments and other research funders about what has been generated from their research investment but requires dedicated resources to complete.

Evaluating the Dissemination and Implementation Impact of a Rehabilitation Intervention: The Graded Repetitive Arm Supplementary Program (GRASP)

Purpose

To evaluate the dissemination and implementation impacts of a rehabilitation intervention.

Methods

Systematic evaluation of data sources including academic publishing metrics, publications, and surveys was used to describe the dissemination and implementation impact of the graded repetitive arm supplementary program (GRASP). Three categories in the Payback Framework were evaluated: knowledge production and dissemination, benefits to future research and research use, and real-world uptake and implementation.

Results

In the Knowledge production and dissemination category, seven publications, authored by the GRASP research team, were associated with the GRASP, and there were approximately 17,000 download counts of GRASP manuals from the website from 120 countries. In the Benefits to future research and research use category, 15 studies and 8 registered clinical trials, authored by researchers outside of the GRASP team, have used GRASP as an intervention. In the real-world uptake and implementation category, GRASP has informed recommendations in 2 clinical guidelines and 20 review papers, and had high implementation uptake (e.g., 35% [53/154] of UK therapists surveyed had used GRASP; 95% [649/681] who downloaded GRASP had used it). More than 75% of those who had used GRASP identified that GRASP provides more intensity in upper extremity rehabilitation, is evidence-based and easy to implement, and the equipment and manual are easy to obtain.

Conclusion

The Payback Framework is useful to evaluate the dissemination and implementation impacts of a rehabilitation intervention. GRASP has been implemented extensively in clinical practice and community in a relatively short time since it has been developed.

On the Frontline-A bibliometric Study on Sustainability, Development, Coronaviruses, and COVID-19

The COVID-19 pandemic has placed the world's population in a state of unprecedented public health and global health vulnerability. Risks to public and global health have escalated due to COVID-19 contamination. This has raised the statistics of inequity and environmental concerns. A possible outlook entails reducing the pandemic consequences by prioritizing development, biodiversity, and adaptability, offering buffer solutions. It contains vital methods for studying, comprehending, and unraveling events-examining early responses to COVID-19, sustainability, and development, relating them with overall Coronaviruses reaction. This study maps out environmental, socioeconomic, and medical/technological issues using as statistical techniques multiple correspondence analysis and validated cluster analysis. The findings encourage rapid, long-term development policy involvement to address the pandemic. The resulting crises have highlighted the necessity for the revival of health justice policies anchored in distinctive public health ethical patterns in response to them. As a general rule, resilience and preparedness will be targeted at developing and vulnerable nations and are prone to include access to vaccines, public health care, and health investment. Our findings show the relevance of innovating on sustainable development routes and yardsticks. Sustainable global health requires crucial measures in prevention, preparation, and response. Long-term policy recommendations are needed to address pandemics and their interrelated crises and foster sustained growth and socioecological protection.

A decision-support tool for funding health innovations at a tertiary academic medical center

OBJECTIVES

To report the processes used to design and implement an assessment tool to inform funding decisions for competing health innovations in a tertiary hospital.

METHODS

We designed an assessment tool for health innovation proposals with three components: "value to the institution," "novelty," and "potential for adoption and scaling." The "value to the institution" component consisted of twelve weighted value attributes identified from the host institution's annual report; weights were allocated based on a survey of the hospital's leaders. The second and third components consisted of open-ended questions on "novelty" and "barriers to implementation" to support further dialogue. Purposive literature review was performed independently by two researchers for each assessment. The assessment tool was piloted during an institutional health innovation funding cycle.

RESULTS

We used 17 days to evaluate ten proposals. The completed assessments were shared with an independent group of panellists, who selected five projects for funding. Proposals with the lowest scores for "value to the institution" had less perceived impact on the patient-related value attributes of "access," "patient centeredness," "health outcomes," "prevention," and "safety." Similar innovations were reported in literature in seven proposals; potential barriers to implementation were identified in six proposals. We included a worked example to illustrate the assessment process.

CONCLUSIONS

We developed an assessment tool that is aligned with local institutional priorities. Our tool can augment the decision-making process when funding health innovation projects. The tool can be adapted by others facing similar challenges of trying to choose the best health innovations to fund.

Uterine Fibroid Embolization or Myomectomy: How Much Marketing Is Enough? Comparative Analysis of Public Search Trends in Google and Medical Publications in PubMed

To compare public popularity and volume of scientific publications regarding uterine fibroid embolization (UFE) and myomectomy. Google Trends and PubMed data were queried to assess temporal variations in online public search volumes and number of research publications for UFE and myomectomy. Time series analysis was used to identify meaningful temporal trends and forecast a future trend. Compared with UFE, myomectomy had significantly higher volumes of public online search and research publications, with an increasing trend over time (P < .0001). The forecasting models predicted a continuing increase in both public search volumes and number of research publications for myomectomy and static future trends in these metrics for UFE. This study signals significantly lower public popularity and research efforts for UFE compared with myomectomy for uterine fibroids. More effective marketing strategies and further research support will be needed to fill this gap.

Retrospective Impact Evaluation Continuing to Prove Challenging Irrespective of Setting: A Study of Research Impact Enablers and Challenges Cloaked as an Impact Evaluation? Comment on "'We're Not Providing the Best Care If We Are Not on the Cutting Edge of Research': A Research Impact Evaluation at a Regional Australian Hospital and Health Service"

The original article provides a detailed and insightful presentation of enablers and detractors for research participation, translation, and impact, at a regional Australian hospital and health service. This information builds on existing knowledge, from the perspective of a non-metropolitan healthcare organisation. It stands to inform all healthcare organisations keen to embed research into their institutions. However, what the article fails to do is present the results of the research impact evaluation in a systematic and useful way for the reader to assess the benefits of research investment by a healthcare organisation including delivery of better quality care and improved patient outcomes. This commentary suggests why such information is critical to justify continued research investment by healthcare organisations and to showcase the potential benefits of the embedded research model. It also discusses the limitations of undertaking impact evaluation retrospectively and suggests that a prospective approach coupled with proper data collection systems and processes upfront could help future reporting of organisational research impact.

Reflections on contributing to health research: A qualitative interview study with research participants and patient advisors

OBJECTIVES

The aims of this study were to explore individuals' experiences of contributing to health research and to identify the types of impact that are perceived as important by participants or patient and public advisors. Specifically, research led by NMAHPP clinicians (Nursing, Midwifery, Allied health professions, Healthcare science, Psychology and Pharmacy).

METHODS

Semi-structured one-to-one interviews were conducted with health research participants and patient or public advisors. Interviewees were recruited from five UK sites and via social media. Interview transcripts were analysed using Thematic Analysis to identify key themes and areas of disagreement.

RESULTS

Twenty-one interviews were completed, and four main themes were identified. The first, optimising research experiences, included personal reflections and broader recommendations to improve participant experiences. The second, connecting health research with healthcare, described research as key for the continued development of healthcare, but illustrated that communication between research teams, participants, and clinicians could be improved. The third theme explored the personal impacts of contributing to research, with interviewees recalling common positive experiences. The final theme discussed capturing research impacts. Interviewees highlighted potential priorities for different stakeholders, but emphasised that financial impacts should not be the sole factor.

CONCLUSION

Individuals who were involved in NMAHPP health research recalled positive experiences and reported good relationships with their research teams. They felt that their contributions were valued. Suggested strategies to optimise the research experience focused on simplifying documentation, clear signposting of the research activities involved, and feedback on the research findings. Routine sharing of relevant research data with clinicians was also recommended. Personal impacts included a deeper understanding of their health condition or health more broadly, and increased confidence interacting with healthcare and other professionals. These findings will be used to inform development of a framework to capture the impact of NMAHPP research.

A research intelligence approach to assess the research impact of the Dutch university medical centres

BACKGROUND

The way in which research impact is evaluated and assessed has long been under debate. In recent years the focus is moving away from the use of numerical indicators, towards an emphasis on narratives. The Dutch university medical centres (UMCs) have a long-standing tradition of using bibliometric indicators. Because of the declining interest in indicators alone, this study was designed to repurpose bibliometrics to answer specific strategic questions. In this article we discuss the strategic and policy-based questions, the methodology we used in uncovering relevant information and conclusions we draw from the analyses we performed. The aim of this article is to inform a broader audience about the potential applications of bibliometric information to support a new form of research intelligence.

METHODS

In this study we used a curated set of publications from the UMCs. We performed different bibliometric analyses and used bibliometric visualization tools to shed light on research focus, open science practices, collaboration, societal impact and scientific impact.

RESULTS

The analyses allowed us to visualize and contextualize the research focus of the UMCs as a whole, but also to show specific focus areas of each UMC. The UMCs are active in the full spectrum of biomedical research, and at the same time are very complementary to each other. Furthermore, we were able to show the development of open access of UMC publications over time, to support the national mission. Visualizing collaboration is a powerful way of showing both the international orientation and the regional and national engine function of UMCs in research. We were able to assess societal impact by looking at the different channels in which publications find their way to societally relevant sources such as news media, policy documents and guidelines. Finally, we assessed scientific impact and put this into an international perspective.

CONCLUSIONS

Research intelligence is able to transform bibliometric information by interpretation and annotation into highly relevant insights that can be used for several different strategic purposes and for research impact assessment in general.

Bibliometric Approach to Evaluating the Impact of a Building Interdisciplinary Research Careers in Women's Health K12 Research Career Development Program

Background: Mentored research career development programs are excellent training opportunities for junior faculty/early-stage investigators to transition into independent research careers. However, there is limited evidence that provides guidance on best practices for measuring the impact and reach of these programs, both for individual Scholars and the program as a whole. This article evaluates both the individual and overall impact of the Building Interdisciplinary Research Careers in Women's Health (BIRCWH) National Institutes of Health research career development award at the University of Minnesota. Materials and Methods: BIRCWH Scholars (n = 16) and a comparison group (n = 17) were evaluated on traditional metrics (e.g., publications, grant funding) in addition to bibliometrics (e.g., network growth, interdisciplinary collaborations, international reach, policy impact). Results: Traditional metric findings showed that BIRCWH Scholars had significantly more publications from pre- to post-BIRCWH experience than the comparison group and more grant funding. Bibliometric findings showed exponential network growth, interdisciplinary collaborations, international citations, and policy impact from pre- to post-BIRCWH Scholar experience. Conclusion: Findings from this evaluation have potential important implications. At the Scholar level, the results can be used to provide evidence of research impact in materials developed for merit review and promotion as well as in job and research grant applications. At the program level, the results can be used at the institutional level to gain broad administrative support and leverage additional funds for program activities and for evidence of program success for continuation funding from federal agencies.

What funders are doing to assess the impact of their investments in health and biomedical research

As pressures to maximize research funding grow, biomedical research funders are increasingly tasked with demonstrating the long-term and real-world impacts of their funded research investments. Over the past three decades, research impact assessments (RIA) have emerged as an important tool for analysing the impacts of research by incorporating logic models, frameworks and indicators to track measures of knowledge production, capacity-building, development of research products, adoption of research into clinical guidelines and policies, and the realization of health, economic and social benefits. While there are currently several models for RIA within the literature, less attention has been paid to how funders can practically select and implement a RIA model to demonstrate the impacts of their own research portfolios. In this paper, a literature review was performed to understand (1) which research funders have performed RIAs of their research portfolios to date; (2) how funders have designed their assessments, including the models and tools they have used; (3) what challenges to and facilitators of success have funders found when adopting the RIA model to their own portfolio; and (4) who participates in the assessments. Forty-four papers from both published and grey literature were found to meet the review criteria and were examined in detail. There is a growing culture of RIA among funders, and included papers spanned a diverse set of funders from 10 countries or regions. Over half of funders (59.1%) used a framework to conduct their assessment, and a variety of methods for collecting impact data were reported. Issues of methodological rigour were observed across studies in the review, and this was related to numerous challenges funders faced in designing timely RIAs with quality impact data. Over a third of articles (36.4%) included input from stakeholders, yet only one article reported surveying patients and members of the public as part of the assessment. To advance RIA among funders, we offer several recommendations for increasing the methodological rigour of RIAs and suggestions for future research, and call for a careful reflection of the voices needed in an impact assessment to ensure that RIAs are having a meaningful impact on patients and the public.

Reflections on recruiting healthcare professionals as research participants: Learning from the ONSPres Study

The involvement of healthcare professionals (HCPs) as research participants is essential to generate high quality evidence for enhancing health services and practice.  Research teams face many challenges in recruiting HCPs for research, and barriers and enablers for interdisciplinary research are not well described in the literature.  The Oral Nutritional Supplement Prescribing Malnutrition Research Study (ONSPres Study) examined malnutrition identification, management, and appropriate oral nutritional supplement prescribing in primary care in Ireland.  The ONSPres Study offers a unique view of recruiting HCPs for research because a range of disciplines were sought for participation in a mixed methods study.  The purpose of this open letter is to describe the experiences of recruitment and participation.  Sixteen general practitioners (GPs) were recruited to participate in one-to-one interviews, eighty health and social care professionals working in community care (including nurses, pharmacists, dietitians, physiotherapists, speech and language therapists, and occupational therapists) were recruited to take part in 12 focus groups, and 31 GPs and trainee GPs were recruited to participate in an education programme developed by the study team.   Strategies required to gain access and reach HCPs differed between disciplines.   Professional networks enhanced access to HCPs working in practice and recruitment was slower and more tailored when those networks were less available to the team.  An interest in malnutrition, to assist in research, to advance patient care, and the opportunity for learning were incentives for the participating HCPs.  Limitations in the diversity of the sample arose, with a bias towards female participants and GPs motivated by an interest in the topic.  It is recommended that study teams collaborate early with relevant HCP disciplines so they can contribute to recruitment planning at project concept and design stages.  To enhance and incentivise HCP participation in research, dedicated time and acknowledgement of participation as continuous professional development is proposed.

Realist evaluation of the impact of the research translation process on health system sustainability: a study protocol

INTRODUCTION

Sustainability at a system level relates to the capacity of the system to be able to service the ongoing health needs of the population. It is a multifaceted concept encompassing both the affordability and efficiency of a system and the system's ability to adapt and change.To address issues that currently threaten health system sustainability, healthcare leaders, policy makers, clinicians and researchers are searching for solutions to ensure the delivery of safe, value-based care into the future. The timely translation of research evidence into sustainable interventions that can be adopted into the health system is one way of bolstering the sustainability of the system as a whole. We present a protocol for the realist evaluation of a research translation funding programme to understand how the research translation process contributes to health system sustainability and value-based healthcare.

METHODS AND ANALYSIS

Underpinned by the realist evaluation framework, we will: (1) Develop the Initial Program Theory (IPT) of the research translation process; (2) Test the program theory through case study analysis; and (3) Refine and consolidate the theory through stakeholder consultation. The evaluation uses a case example of a research translation programme, chosen for its representation of a microcosm of the broader health system and the heterogeneity of service improvement activities taking place within it. Across the three phases, analysis of data from documents about the research translation program and interviews and focus groups with stakeholders and program users will draw on the context (C), mechanism (M), outcome (O) formula that is core to realist evaluation. In addition, system dynamic methods will capture the feedback loops and complex relationships among the IPT and context-mechanism-outcome configurations. This approach to evaluation of a research translation funding programme may be adapted to similar programmes operating in other settings.

ETHICS AND DISSEMINATION

Curtin University Human Research Ethics Committee, Western Australia, approved this study (approval number: HRE2020-0464). Results will be published in scientific journals, and communicated to respondents and relevant partners.

Collective health research assessment: developing a tool to measure the impact of multistakeholder research initiatives

Background

The need to more collaboratively measure the impact of health research and to do so from multidimensional perspectives has been acknowledged. A scorecard was developed as part of the Collective Research Impact Framework (CRIF), to engage stakeholders in the assessment of the impacts of health research and innovations. The purpose of this study was to describe the developmental process of the MULTI-ACT Master Scorecard (MSC) and how it can be used as a workable tool for collectively assessing future responsible research and innovation measures.

Methods

An extensive review of the health research impact literature and of multistakeholder initiatives resulted in a database of 1556 impact indicators. The MSC was then cocreated by engaging key stakeholders and conducting semi-structured interviews of experts in the field.

Results

The MSC consists of five accountability dimensions: excellence, efficacy, economic, social and patient-reported outcomes. The tool contains 125 potential indicators, classified into 53 impact measurement aspects that are considered the most relevant topics for multistakeholder research and innovation initiatives when assessing their impact on the basis of their mission and their stakeholders’ interests. The scorecard allows the strategic management of multistakeholder research initiatives to demonstrate their impact on people and society. The value of the tool is that it is comprehensive, customizable and easy to use.

Conclusions

The MSC is an example of how the views of society can be taken into account when research impacts are assessed in a more sustainable and balanced way. The engagement of patients and other stakeholders is an integral part of the CRIF, facilitating collaborative decision-making in the design of policies and research agendas. In policy making, the collective approach allows the evaluation perspective to be extended to the needs of society and towards responsible research and innovation. Multidimensionality makes research and innovations more responsive to systemic challenges, and developing more equitable and sustainable health services.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12961-022-00856-9.

Research Metrics for Health Science Schools: A Conceptual Exploration and Proposal

Research is a critical component of the public health enterprise, and a key component of universities and schools of public health and medicine. To satisfy varying levels of stakeholders in the field of public health research, accurately measuring the return on investment (ROI) is important; unfortunately, there is no approach or set of defined metrics that are universally accepted for such assessment. We propose a research metrics framework to address this gap in higher education. After a selected review of existing frameworks, we identified seven elements of the generic research lifecycle (five internal to an institution and two external). A systems approach was then used to broadly define four parts of each element: inputs, processes, outputs, and outcomes (or impacts). Inputs include variables necessary to execute research activities such as human capital and finances. Processes are the pathways of measurement to track research performance through all phases of a study. Outputs entail immediate products from research; and outcomes/impacts demonstrate the contribution research makes within and beyond an institution. This framework enables the tracking and measurement of research investments to outcomes. We acknowledge some of the challenges in applying this framework including the lack of standardization in research metrics, disagreement on defining impact among stakeholders, and limitations in resources for implementing the framework and collecting relevant data. However, we suggest that this proposed framework is a systematic way to raise awareness about the role of research and standardize the measurement of ROI across health science schools and universities.

An impact review of a Western Australian research translation program

The translation gap between knowledge production and implementation into clinical practice and policy is an ongoing challenge facing researchers, funders, clinicians and policy makers globally. Research generated close to practice and in collaboration with end users is an approach that is recognised as an effective strategy to facilitate an improvement in the relevance and use of health research as well as building research capacity amongst end users. The Research Translation Projects (RTP) program funded by the Western Australian (WA) Department of Health facilitates clinical and academic collaboration through competitive funding of short-term research projects. Its aim is to improve healthcare practice while also finding efficiencies that can be delivered to the WA health system. A mixed methods approach was adopted to evaluate the research impact of the RTP program, at completion of the two-year funding period, across a range of impact domains through the adaptation and application of the Canadian Academy of Health Sciences’ (CAHS) framework for research impact. In addition, further analysis was undertaken to address specific objectives of the RTP program more closely, in particular research capacity building and collaboration and health system Inefficiencies targeted by the program. Social network analysis was applied to assess the extent and growth of collaboration across WA health organisations over time. Results indicated that the ‘bottom up’ approach to research translation has triggered modest, yet positive outcomes across impact domains including advancing knowledge, collaboration and capacity building as well as contributing to changes in policy and practice. Additionally, the projects identified opportunities by which inefficiencies in the health system can be addressed. Further work is required to better understand the pathways by which short-term outcomes can be translated into more long-term impacts and the mechanisms that trigger this process.

Recent trends in the use of social media in parasitology and the application of alternative metrics

In recent times, the use of social media for the dissemination of “news and views” in parasitology has increased in popularity. News, Twitter and Blogs have emerged as commonplace vehicles in the knowledge dissemination and transfer process. Alternative metrics (“altmetrics”), based on social media mentions have been proposed as a measure of societal impact, although firm evidence for this relationship is yet to be found. Nevertheless, increasing amounts of data on “altmetrics” are being analysed to identify the nature of the unknown impact that social media is generating. Here, we examine the recent, and increasing use of social media in the field of parasitology and the relationship of “altmetrics” with more traditional bibliometric indicators, such as article citations and journal metrics. The analyses document the rise and dominance of Twitter as the main form of social media occurring in the discipline of parasitology and note the contribution to this trend of Twitter bots that automatically tweet about publications. We also report on the use of the social referencing platform Mendeley and its correlation to article citations; Mendeley reader numbers are now considered to provide firm evidence on the early impact of research. Finally, we consider the Twitter profile of 31 journals publishing parasitology research articles (by volume of papers published); we show that 13 journals are associated with prolific Twitter activity about parasitology. We hope this study will stimulate not only the continued and responsible use of social media to disseminate knowledge about parasitology for the greater good, but also encourage others to further investigate the impact and benefits that altmetrics may bring to this discipline.

•

We highlight and document the rise of social media and its use in parasitology.

•

Twitter activity within the parasitology community has increased significantly over the last 10 years.

•

Mendeley reader activity is strongly correlated with an article's citations.

•

Thirteen journals are associated with prolific Twitter activity about parasitology.

•

A Journalʼs social media strategy is important to authors.

Funding patterns for biomedical research and infectious diseases burden in Gabon

Background

Biomedical research plays an important role in improving health. There seems to exist a negative correlation between the amount of biomedical research funding and disease burden from all Sub-Saharan African countries. In this study, we describe funding patterns for biomedical research, explore the correlation between funding and burden of diseases, and quantify inequalities in funds distribution across diseases in Gabon over the period 2005–2015.

Methods

Data on medical research funds from 2005 to 2015 were retrieved through a structured questionnaire distributed to Gabonese biomedical research institutions and by consulting online databases. Data on the burden of diseases were gathered from the World Health Organization and the Institute for Health Metrics and Evaluation. We used Kendall rank correlation coefficient to explore the correlation between cumulative funds over time and the burden of disease. The inequality distribution of funding across diseases was assessed through Gini coefficient and Lorenz curve.

Results

Biomedical research funding was characterized by a remarkable growth from 2005 to 2010 and a decline from 2010 to 2014. Funds were mostly from external sources and from partnerships. There was inequality in research funds allocation across diseases and malaria was far the most funded disease. There was a significant negative correlation between cumulative funding and the burden of HIV, tuberculosis, and of Helminthiasis (from 2006 to 2010) suggesting that research may be contributing to the management of such diseases. A positive, although not significant, correlation was found between cumulative funds and malaria burden.

Conclusions

The negative correlation between HIV and tuberculosis cumulative funding and burden suggests that research may be contributing to the management of such diseases but further research is needed to assess the causal direction of such as relationship. As the burden of non-communicable diseases is increasing, more research funds should be focused on those. While research partnerships have been and will remain fundamental, Gabon should increase the amount of national funds to overcome periods of reduced research funding flows from abroad.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12889-021-12201-w.

Improving Food Systems: A Participatory Consultation Exercise to Determine Priority Research and Action Areas in Viet Nam

With increased burden of malnutrition on global health, there is a need to set clear and transparent priorities for action in food systems at a global and local level. While priority settings methods are available for several adjacent domains, such as nutrition and health policies, setting priorities for food system research has not been documented and streamlined. The challenges involve food systems' multisector, multi-stakeholder and multi-outcome nature. Where data exists, it is not easy to aggregate data from across food system dimensions and stakeholders to make an informed analysis of the overall picture of the food system, as well as current and potential food system trade-offs to inform research and policy. Once research priorities are set, they risk staying on paper and never make their ways to concrete outputs and outcomes. In this paper, we documented and assessed the inclusive process of setting research priorities for a local food system, taking Vietnamese food systems as a case study. From this exercise, we examined how priority setting for food systems research could learn from and improve upon earlier priority setting research practices in other domains. We discussed the lessons for research and policies in local food systems, such as the need for a concrete follow-up plan accompanying the priority setting process.

It takes two to tango: Examining productive interactions in urban research collaboration

The science–society relations of social sciences and humanities have been increasingly discussed under the concept of productive interactions, which refers to the mutual learning processes between researchers and stakeholders for the benefit of societal development. While most studies have analysed the societal impact from the research performers’ perspective, in this article, we examine the practitioners’ side. We contribute to the evaluation theory by offering a new perspective to examine the emergence of productive interactions. Based on an empirical analysis of collaborative practices in two Finnish urban research programmes and how the practitioners reflected on them, we argue that practitioners’ competencies are essential in leveraging societal impact. The improvement of these ‘pracademic competencies’ need to be raised as an issue in research policy and evaluation promoting responsible research and innovation.

Impacts of clinical academic activity: qualitative interviews with healthcare managers and research-active nurses, midwives, allied health professionals and pharmacists

OBJECTIVES

To explore the perceived impacts of clinical academic activity among the professions outside medicine.

DESIGN

Qualitative semistructured interviews.

SETTING AND PARTICIPANTS

There were two groups of interviewees: Research-active nurses, midwives, allied health professionals, healthcare scientists, psychologists and pharmacists (NMAHPPs) and managers of these professions. All participants were employed in a single, multisite healthcare organisation in the UK.

ANALYSIS

Interview transcripts were analysed using the framework method to identify key themes, subthemes and areas of divergence.

RESULTS

Four themes were identified. The first, cultural shifts, described the perceived improvements in the approach to patient care and research culture that were associated with clinical academic activity. The second theme explored visibility and included the positive reputation that clinical academics were identified as bringing to the organisation in contrast with perceived levels of invisibility and inaccessibility of these roles. The third theme identified the impacts of the clinical academic pathways, including the precarity of these roles. The final theme explored making impact tangible, and described interviewees' suggestions of possible methods to record and demonstrate impact.

CONCLUSIONS

Perceived positive impacts of NMAHPP clinical academic activity focused on interlinked positive changes for patients and clinical teams. This included delivery of evidence-based healthcare, patient involvement in clinical decision making and improved staff recruitment and retention. However, the positive impacts of clinical academic activity often centred around individual clinicians and did not necessarily translate throughout the organisation. The current clinical academic pathway was identified as causing tension between the perceived value of clinical academic activity and the need to find sufficient staffing to cover clinical services.

Evaluation of the public health achievements made by projects supported by a federal contract mechanism at the Centers for Disease Control and Prevention (CDC), USA

In 2012, the Centers for Disease Control and Prevention (CDC) established the Achieving Public Health Impact through Research (APHIR) contract mechanism. APHIR provides CDC's Centers, Institute, and Offices (CIOs) a mechanism that supports multiyear, high impact public health research. Awarded projects supported research on a wide range of topics (e.g., cancer surveillance, HIV education programs, development of biological assays, and evaluation of traumatic brain injury prevention programs) and achieved diverse outcomes (e.g., contribution to the body of knowledge in their field, changes in practice and health service delivery, and capacity building). This article describes how existing impact frameworks and a variety of methods and tools (key informant interviews, online survey, bibliometric analysis, Altmetric and document reviews) were used to identify the outcomes achieved by awarded projects. The approach discussed in this paper can be used to evaluate projects that involve a diversity of activities and outcomes.

The Corona-Eye: Exploring the risks of COVID-19 on fair assessments of impact for REF2021

This paper assesses the risk of two COVID-19 related changes necessary for the expert-review of the REF2021’s Impact criterion: the move from F2F to virtual deliberation; and the changing research landscape caused by the COVID-19 crisis requiring an extension of deadlines, and accommodation of COVID-19 related mitigation. Peer review in its basic form requires expert debate, where dissenting opinions and non-verbal cues are absorbed into a groups deliberative practice and therefore inform outcomes. With a move to deliberations in virtual settings, the most likely current outcome for REF2021 evaluations, the extent that negotiation dynamics necessary in F2F evaluations are diminished and how this limits panelists’ ability to sensitively assess COVID-19 mitigation statements is questioned. This article explores the nature of, and associated capabilities to undertake, complex decision making in virtual settings around the Impact criterion as well the consequences of COVID-19 on normal Impact trajectories. It examines the risks these changes present for evaluation of the Impact criterion and provides recommendations to offset these risks to enhance discussion and safeguard the legitimacy of evaluation outcomes. This paper is also relevant for evaluation processes of academic criteria that require both a shift to virtual, and/or guidance of how to sensitively assess the effect of COVID-19 on narratives of individual, group or organisational performance.

Directing Research Toward Health Equity: a Health Equity Research Impact Assessment

Despite medical research advancements, inequities persist, as research has enhanced the health of some while leaving many communities untouched. Reforms are needed to direct research toward health equity, both during this pandemic and beyond. All research must currently pass scientific and ethical review processes, but neither may adequately examine a project's potential impact on inequities and local communities. Research stakeholders need practical tools to help review and examine any given study's impact on health equity. We articulate a health equity research impact assessment, which draws from existing research impact assessments and health disparities research measures and frameworks. We describe how this tool was developed and how it may be used by research reviewers, researchers, academic institutions, and funding agencies to elevate health equity in medical science.

Exploring research impact models: A systematic scoping review

This article explores the models and frameworks developed on “research impact’. We aim to provide a comprehensive overview of related literature through scoping study method. The present research investigates the nature, objectives, approaches, and other main attributes of the research impact models. It examines to analyze and classify models based on their characteristics. Forty-seven studies and 10 reviews published between 1996 and 2020 were included in the analysis. The majority of models were developed for the impact assessment and evaluation purposes. We identified three approaches in the models, namely outcome-based, process-based, and those utilized both of them, among which the outcome-based approach was the most frequently used by impact models and evaluation was considered as the main objective of this group. The process-based ones were mainly adapted from the W.K. Kellogg Foundation logic model and were potentially eligible for impact improvement. We highlighted the scope of processes and other specific features for the recent models. Given the benefits of the process-based approach in enhancing and accelerating the research impact, it is important to consider such approach in the development of impact models. Effective interaction between researchers and stakeholders, knowledge translation, and evidence synthesis are the other possible driving forces contributing to achieve and improve impact.

Scholarly Activity of Radiation Oncologists in High-Income Developing Countries: Saudi Arabia as an Example

PURPOSE

To examine the trends and quality metrics of publications by radiation oncologists in Saudi Arabia.

METHODS

PubMed was searched using names of all Saudi radiation oncologists to retrieve published articles between January 2010 and December 2019. International collaboration, journal impact factor and country of origin, and number of citations were collected. Each article was assessed for epidemiologic type and independently assigned a level of evidence (LOE) by two authors. The trend in publications was examined and compared in the first and second 5-year periods (2010-2014 and 2015-2019) using relevant parameters.

RESULTS

A total of 186 publications were found and included. The most common type of research was cohort studies followed by case reports and case series in 24%, 14%, and 13% of all publications, respectively. Dosimetry, clinical, and preclinical studies formed 7%, 8.6%, and 7.5% of the total publications, respectively. The LOE was I, II, III, IV, and not applicable in 8.6%, 22%, 25.8%, 29%, and 14.5% of the included publications, respectively. Comparing the first and second 5-year periods, there was an increase in international collaboration (P < .001) in the second period. The number of citations (P < .001) and journal impact factor (P = .028) were lower in the second period. LOE and publications in international journals were not statistically different between the two periods.

CONCLUSION

Although radiation oncology research activity in Saudi Arabia has gained momentum in terms of volume and international collaboration over time, the LOE has not improved. This calls for a national effort to make the contribution to the literature a priority, allocate adequate resources, and apply appropriate measures to enhance research productivity and quality.

Translation and Impact of Funded Australian Cardiovascular Research: A Review With Perspective

This aim of this paper is to set the scene for the need for impact assessment and return on investment in funded cardiovascular research in Australia, starting with the historical perspective on waste in health and medical research. Recently there has been a substantial move from discussion and policy about the need for research translation, into practice and application via the evolution of funding streams like the Australian Medical Research Future Fund (MRFF). Health and medical research funders play a critical role in both setting the expectations for research translation and impact and helping researchers to meet these expectations. As a leading cause of death, cardiovascular disease is a national health priority, recognised as such with a AUD$220 million MRFF allocation to the Cardiovascular Health Mission. Focussing on cardiovascular research, we address some of the barriers researchers face in prospectively planning for research translation and impact assessment, and call for an ecosystem that supports a return on investment for all stakeholders, especially the community and patient end-users.

A decade of Ecuador´s efforts to raise its health research output: a bibliometric analysis

Background: Over the past decade, the political movement called ‘Revolución Ciudadana’ implemented a variety of policies and interventions (P&I) in Ecuador to improve higher education and strengthen local research capacity. We refer specifically to the ‘Mandato 14’ and the Higher Education Law (LOES, Spanish acronym) launched in 2008 and 2010, respectively.

Objective: To assess the impact of these P&I (Mandato 14/LOES) on the production of health sciences-related articles (HSRA), and the relationship of these HSRA with the country’s health priorities.

Methods: A Scopus search was performed to retrieve HSRA published from 1999 to 2017. Bivariate analysis was used to assess variation between the period I (1999–2008) and period II (2009–2017). Further, we examined the association between the top 10 causes of mortality and the total HSRA output.

Results: The final study sample consisted of 2784 articles. After 2008, Ecuadorian production of HSRA increased steadily from 671 to 2133 publications (p<.001). Overall (1999–2017), the most common study design was cross-sectional (32.3%), the primary research focus was in the clinical-surgical area (49.3%), and the academic institutions were the primary drivers of scientific production during period II (56.9% vs. 29.5%, p<.001). Further, we found a decrease in the production of randomized controlled trials (6.7% vs. 1.8%, p<.001). Only 9% of research production involved the primary causes of mortality, and the proportion has remained unchanged over time (8.2% vs. 9.3%, p>.05).

Conclusions: Ecuadorian HSRA output increased significantly after 2008. This larger volume of scientific output could be the result to the Mandato 14/LOES implemented in the last decade. However, a low percentage of HSRA are dedicated to addressing the country’s health priorities. Proper planning, execution and monitoring of national health research agendas would reduce the mismatch between health burden and the HSRA output in Ecuador and other low-and middle-income countries.

A qualitative systematic review and thematic synthesis exploring the impacts of clinical academic activity by healthcare professionals outside medicine

BACKGROUND

There are increasing opportunities for healthcare professionals outside medicine to be involved in and lead clinical research. However, there are few roles within these professions that include time for research. In order to develop such roles, and evaluate effective use of this time, the range of impacts of this clinical academic activity need to be valued and understood by healthcare leaders and managers. To date, these impacts have not been comprehensively explored, but are suggested to extend beyond traditional quantitative impact metrics, such as publications, citations and funding awards.

METHODS

Ten databases, four grey literature repositories and a naïve web search engine were systematically searched for articles reporting impacts of clinical academic activity by healthcare professionals outside medicine. Specifically, this did not include the direct impacts of the research findings, rather the impacts of the research activity. All stages of the review were performed by a minimum of two reviewers and reported impacts were categorised qualitatively according to a modified VICTOR (making Visible the ImpaCT Of Research) framework.

RESULTS

Of the initial 2704 identified articles, 20 were eligible for inclusion. Identified impacts were mapped to seven themes: impacts for patients; impacts for the service provision and workforce; impacts to research profile, culture and capacity; economic impacts; impacts on staff recruitment and retention; impacts to knowledge exchange; and impacts to the clinical academic.

CONCLUSIONS

Several overlapping sub-themes were identified across the main themes. These included the challenges and benefits of balancing clinical and academic roles, the creation and implementation of new evidence, and the development of collaborations and networks. These may be key areas for organisations to explore when looking to support and increase academic activity among healthcare professionals outside medicine. The modified VICTOR tool is a useful starting point for individuals and organisations to record the impact of their research activity. Further work is needed to explore standardised methods of capturing research impact that address the full range of impacts identified in this systematic review and are specific to the context of clinical academics outside medicine.

Applying a framework to assess the impact of cardiovascular outcomes improvement research

Background

Health and medical research funding agencies are increasingly interested in measuring the impact of funded research. We present a research impact case study for the first four years of an Australian National Health and Medical Research Council funded Centre of Research Excellence in Cardiovascular Outcomes Improvement (2016–2020). The primary aim of this paper was to explore the application of a research impact matrix to assess the impact of cardiovascular outcomes improvement research.

Methods

We applied a research impact matrix developed from a systematic review of existing methodological frameworks used to measure research impact. This impact matrix was used as a bespoke tool to identify and understand various research impacts over different time frames. Data sources included a review of existing internal documentation from the research centre and publicly available information sources, informal iterative discussions with 10 centre investigators, and confirmation of information from centre grant and scholarship recipients.

Results

By July 2019, the impact on the short-term research domain category included over 41 direct publications, which were cited over 87 times (median journal impact factor of 2.84). There were over 61 conference presentations, seven PhD candidacies, five new academic collaborations, and six new database linkages conducted. The impact on the mid-term research domain category involved contributions towards the development of a national cardiac registry, cardiovascular guidelines, application for a Medicare Benefits Schedule reimbursement item number, introduction of patient-reported outcome measures into several databases, and the establishment of nine new industry collaborations. Evidence of long-term impacts were described as the development and use of contemporary management for aortic stenosis, a cardiovascular risk prediction model and prevention targets in several data registries, and the establishment of cost-effectiveness for stenting compared to surgery.

Conclusions

We considered the research impact matrix a feasible tool to identify evidence of academic and policy impact in the short- to midterm; however, we experienced challenges in capturing long-term impacts. Cost containment and broader economic impacts represented another difficult area of impact to measure.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12961-021-00710-4.

Evaluating cancer research impact: lessons and examples from existing reviews on approaches to research impact assessment

BACKGROUND

Performing cancer research relies on substantial financial investment, and contributions in time and effort from patients. It is therefore important that this research has real life impacts which are properly evaluated. The optimal approach to cancer research impact evaluation is not clear. The aim of this study was to undertake a systematic review of review articles that describe approaches to impact assessment, and to identify examples of cancer research impact evaluation within these reviews.

METHODS

In total, 11 publication databases and the grey literature were searched to identify review articles addressing the topic of approaches to research impact assessment. Information was extracted on methods for data collection and analysis, impact categories and frameworks used for the purposes of evaluation. Empirical examples of impact assessments of cancer research were identified from these literature reviews. Approaches used in these examples were appraised, with a reflection on which methods would be suited to cancer research  impact evaluation going forward.

RESULTS

In total, 40 literature reviews were identified. Important methods to collect and analyse data for impact assessments were surveys, interviews and documentary analysis. Key categories of impact spanning the reviews were summarised, and a list of frameworks commonly used for impact assessment was generated. The Payback Framework was most often described. Fourteen examples of impact evaluation for cancer research were identified. They ranged from those assessing the impact of a national, charity-funded portfolio of cancer research to the clinical practice impact of a single trial. A set of recommendations for approaching cancer research impact assessment was generated.

CONCLUSIONS

Impact evaluation can demonstrate if and why conducting cancer research  is worthwhile. Using a mixed methods, multi-category assessment organised within a framework, will provide a robust evaluation, but the ability to perform this type of assessment may be constrained by time and resources. Whichever approach is used, easily measured, but inappropriate metrics should be avoided. Going forward, dissemination of the results of cancer research impact assessments will allow the cancer research community to learn how to conduct these evaluations.

[Research impact and participatory health research: an international debate]

Seit einigen Jahren beschäftigt sich die akademische Welt verstärkt damit, welcher gesellschaftliche Nutzen von Forschung ausgeht und wie er erhoben und dargestellt werden kann. Zu der Wirkung von Forschung, dem sogenannten Forschungsimpact, werden verschiedene Diskussionen geführt – kontrovers besonders in Ländern, in denen Impact mittlerweile ein Faktor bei der Verteilung von Fördergeldern geworden ist. Partizipative Gesundheitsforschung ist ein Forschungsansatz, der diejenige in die Forschung einbindet, deren Arbeit oder Leben im Forschungsinteresse stehen. Mit diesem Ansatz soll auch außerhalb der akademischen Welt Wirkung erzielt werden, um Veränderungen zum gesundheitlichen Wohlergehen anzustoßen und die gesundheitliche Chancengleichheit in der Gesellschaft zu erhöhen.

Der Beitrag geht den Fragen nach, wie Forschungsimpact verstanden und erhoben werden kann und welchen Beitrag die Wirkungen der partizipativen Gesundheitsforschung zu gesundheitlicher Chancengleichheit leisten können. Er geht exemplarisch auf Aspekte von Impactdiskussionen im Vereinigten Königreich und in Deutschland ein und bildet dann den aktuellen Stand der fachlichen Auseinandersetzung mit dem Themenbereich ab. Schließlich beschreibt er die Wirkfaktoren der partizipativen Gesundheitsforschung, die zu einer Stärkung gesundheitlicher Chancengleichheit führen können.

Nicht nur im Bereich der partizipativen Gesundheitsforschung ermöglicht es die Diskussion über Forschungsimpact, die Nachhaltigkeit und den Wert von Forschung zu bewerten und zu stärken. Da die vielseitigen Wirkungen der partizipativen Gesundheitsforschung jedoch das Potenzial haben, gesundheitliche Ungleichheiten zu verringern, sollten sie entsprechend wahrgenommen und anerkannt werden.

Articulating the "So, What?" in Clinical Research: Insight from the M-CHOIR Group

With the academic culture of "publish or perish," authors must ensure that they are delivering high-quality data with a meaningful impact on clinical practice. Even for physician-scientists at the top of their fields, establishing the relevance of a study to clinical practice is a challenge. Thus, it is essential that research proposals ask questions that are clinically important, use appropriate methodologies, and examine outcomes that are relevant to both the physicians and the patients. The question of "so, what?" or in other words, "who cares?" is one that can make or break a study's impact on clinical practice. Researchers should use models such as PICOS (Population, Intervention, Comparison, Outcomes, and Study design) and FINER (Feasible, Interesting, Novel, Ethical, Relevant) and ask why readers will care about their study's findings before the study is conducted. By doing so, researchers can ensure the successful execution of their study and a meaningful impact of their findings, in both academia and clinical practice. This Special Topic article aims to guide researchers in producing relevant, impactful conclusions of their studies by providing input and resources from the Michigan Center for Hand Outcomes and Innovation (M-CHOIR) group.

Stroke incidence and subtypes in Aboriginal people in remote Australia: a healthcare network population-based study

OBJECTIVES

We aimed to compare the incidence, subtypes and aetiology of stroke, and in-hospital death due to stroke, between Aboriginal and non-Aboriginal people in Central Australia, a remote region of Australia where a high proportion Aboriginal people reside (40% of the population). We hypothesised that the rates of stroke, particularly in younger adults, would be greater in the Aboriginal population, compared with the non-Aboriginal population; we aimed to elucidate causes for any identified disparities.

DESIGN

A retrospective population-based study of patients hospitalised with stroke within a defined region from 1 January 2011 to 31 December 2014.

SETTING

Alice Springs Hospital, the only neuroimaging-capable acute hospital in Central Australia, serving a network of 50 healthcare facilities covering 672 000 km².

PARTICIPANTS

161 residents (63.4% Aboriginal) of the catchment area admitted to hospital with stroke.

PRIMARY AND SECONDARY OUTCOME MEASURES

Rates of first-ever stroke, overall (all events) stroke and in-hospital death.

RESULTS

Of 121 residents with first-ever stroke, 61% identified as Aboriginal. Median onset-age (54 years) was 17 years younger in Aboriginal patients (p<0.001), and age-standardised stroke incidence was threefold that of non-Aboriginal patients (153 vs 51 per 100 000, incidence rate ratio 3.0, 95% CI 2 to 4). The rate ratios for the overall rate of stroke (first-ever and recurrent) were similar. In Aboriginal patients aged <55 years, the incidence of ischaemic stroke was 14-fold greater (95% CI 4 to 45), and intracerebral haemorrhage 19-fold greater (95% CI 3 to 142) than in non-Aboriginal patients. Crude prevalence of diabetes mellitus (70.3% vs 34.0%, p<0.001) and hypercholesterolaemia (68.9% vs 51.1%, p=0.049) was greater, and age-standardised in-hospital deaths were fivefold greater (35 vs 7 per 100 000, 95% CI 2 to 11) in Aboriginal patients than in non-Aboriginal patients.

CONCLUSIONS

Stroke incidence (both subtypes) and in-hospital deaths for remote Aboriginal Australians are dramatically greater than in non-Aboriginal people, especially in patients aged <55 years.