Evaluating impact from research: A methodological framework

Diet and health inequalities: Connecting with vulnerable groups to address food insecurity-the DIO food project

The current cost-of-living crisis is disproportionately affecting families experiencing poverty and is likely to be amplifying existing dietary inequalities and challenges, such as food insecurity (FI). Government policies designed to address diet inequality in the UK have historically had minimal impact on population diet and health and may have even widened existing inequalities. Therefore, the effect of nutrition policies on those experiencing FI in the context of the current cost-of-living crisis needs to be better understood. The aim of the Diet and Health Inequalities (DIO Food) project is to work with early years, people living on a low income and retailers to generate opportune evidence-based research and commentary that will inform diet-related health inequalities policy and practice. DIO Food is related to the existing Food Insecurity in people living with Obesity (FIO Food) project, which consists of four interlinked work packages (WPs1-4). DIO Food consists of three interlinked work packages (WPs5-7), which enhance the scope of FIO Food, and are described in this article. WP5 addresses a paucity of research on maternal and infant food insecurity in the UK, by applying a qualitative research approach to capture parents' and carers' perceptions of the relationship between the food system and other influences impacting infant feeding practice, associated with the cost-of-living crisis. WP6 will conduct the first-ever cross-retailer independent evaluation of England's high fat, sugar and salt (HFSS) product placement legislation. Researchers will analyse store-level supermarket sales data provided by large UK retailers to produce sector-level insights into whether HFSS legislation reduced HFSS purchasing, improved the healthiness of retailer product portfolios, and was equitable across areas with different characteristics. WP7 will support WP5 and 6, by strengthening engagement with key stakeholders, including at-risk consumers and representatives of major supermarkets, and effectively translating research outcomes and stakeholder perspectives for policy and industry decision-makers.

The LTAR Cropland Common Experiment at R. J. Cook Agronomy Farm

Dryland agriculture in the Inland Pacific Northwest is challenged in part by rising input costs for seed, fertilizer, and agrichemicals; threats to water quality and soil health, including soil erosion, organic matter decline, acidification, compaction, and nutrient imbalances; lack of cropping system diversity; herbicide resistance; and air quality concerns from atmospheric emissions of particulate matter and greenhouse gases. Technological advances such as rapid data acquisition, artificial intelligence, cloud computing, and robotics have helped fuel innovation and discovery but have also further complicated agricultural decision-making and research. Meeting these challenges has promoted interest in (1) supporting long-term research that enables assessment of ecosystem service trade-offs and advances sustainable and regenerative approaches to agriculture, and (2) developing coproduction research approaches that actively engage decision-makers and accelerate innovation. The R. J. Cook Agronomy Farm (CAF) Long-Term Agroecosystem Research (LTAR) site established a cropping systems experiment in 2017 that contrasts prevailing (PRV) and alternative (ALT) practices at field scales over a proposed 30-year time frame. The experimental site is on the Washington State University CAF near Pullman, WA. Cropping practices include a wheat-based cropping system with wheat (Triticum aestivum L.), canola (Brassica napus, variety napus), chickpea (Cicer arietinum), and winter pea (Pisum sativum), with winter wheat produced every third year under the ALT practices of continuous no-tillage and precision applied N, compared to the PRV practice of reduced tillage (RT) and uniformly applied agrichemicals. Biophysical measurements are made at georeferenced locations that capture field-scale spatial variability at temporal intervals that follow approved methods for each agronomic and environmental metric. Research to date is assessing spatial and temporal variations in cropping system performance (e.g., crop yield, soil health, and water and air quality) for ALT versus PRV and associated tradeoffs. Future research will explore a coproduction approach with the intent of advancing discovery, innovation, and impact through collaborative stakeholder-researcher partnerships that direct and implement research priorities.

Changes in Health Spending During the COVID-19 Pandemic by Expenditure Type and Household Profiles in Mexican Households

OBJECTIVE

This study aimed to analyze the changes in the health expenditure of households in Mexico during the COVID-19 pandemic to approximate changes in healthcare that can lead to difficulties in detecting noncommunicable diseases, among others.

METHODS

We compare health spending before and after the pandemic through various estimators using multivariate linear regression models at the household level. We also explore heterogeneous effects by zone, sex, and household composition by age. We explore potential mechanisms of change estimating probit models of healthcare. We use microdata from the National Health and Nutrition Survey 2018 and COVID-19.

RESULTS

The results suggest a significant reduction in health spending, mainly referring to oral health, clinical analysis, and medical studies. Moreover, differences arise by type of area and household age profile. Changes are more significant among families with children younger than 12 years and households situated in urban areas. Regarding the mechanisms, the results suggest that the lower spending is not due to fewer health needs but rather due to less healthcare attention.

CONCLUSION

The COVID-19 pandemic had a significant and heterogeneous impact on household health spending. This lower spending could lead to less detection of noncommunicable diseases, translating into more pressure on the health system in the medium and long term.

Evaluating coupling coordination between urban smart performance and low-carbon level in China's pilot cities with mixed methods

The construction models of smart cities and low-carbon cities are crucial for advancing global urbanization, enhancing urban governance, and addressing major urban challenges. Despite significant advancements in smart and low-carbon city research, a consensus on their coupling coordination remains elusive. This study employs mixed-method research, combining qualitative and quantitative analyses, to investigate the coupling coordination between urban smart performance (SCP) and low-carbon level (LCL) across 52 typical smart and low-carbon pilot cities in China. Independent evaluation models for SCP and LCL qualitatively assess the current state of smart and low-carbon city construction. Additionally, an Entropy-TOPSIS-Pearson correlation-Coupling coordination degree (ETPC) analysis model quantitatively examines their relationship. The results reveal that smart city initiatives in China significantly outperform low-carbon city development, with notable disparities in SCP and LCL between eastern, non-resource-based, and central cities versus western, resource-dependent, and peripheral cities. A strong positive correlation exists between urban SCP and overall LCL, with significant correlations in management, society, and economy, and moderate to weak correlations in environmental quality and culture. As SCP levels improve, the coupling coordination degree between the urban SCP and LCL systems also increases, driven primarily by economic, management, and societal factors. Conversely, the subsystems of low-carbon culture and environmental quality show poorer integration. Based on these findings, this study proposes an evaluation system for smart and low-carbon coupling coordination development, outlining pathways for future development from the perspective of urban complex systems.

Evaluating the perceived impact and legacy of master's degree level research in the allied health professions: a UK-wide cross-sectional survey

BACKGROUND

Post graduate master's degree qualifications are increasingly required to advance allied health profession careers in education, clinical practice, leadership, and research. Successful awards are dependent on completion of a research dissertation project. Despite the high volume of experience gained and research undertaken at this level, the benefits and impact are not well understood. Our study aimed to evaluate the perceived impact and legacy of master's degree training and research on allied health profession practice and research activity.

METHODS

A cross-sectional online survey design was used to collect data from allied health professionals working in the United Kingdom who had completed a postgraduate master's degree. Participants were recruited voluntarily using social media and clinical interest group advertisement. Data was collected between October and December 2022 and was analysed using descriptive statistics and narrative content analysis. Informed consent was gained, and the study was approved by the university research ethics committee.

RESULTS

Eighty-four responses were received from nine allied health professions with paramedics and physiotherapists forming the majority (57%) of respondents. Primary motivation for completion of the master's degree was for clinical career progression (n = 44, 52.4%) and formation of the research dissertation question was predominantly sourced from individual ideas (n = 58, 69%). Formal research output was low with 27.4% (n = 23) of projects published in peer reviewed journal and a third of projects reporting no output or dissemination at all. Perceived impact was rated highest in individual learning outcomes, such as improving confidence and capability in clinical practice and research skills. Ongoing research engagement and activity was high with over two thirds (n = 57, 67.9%) involved in formal research projects.

CONCLUSION

The focus of master's degree level research was largely self-generated with the highest perceived impact on individual outcomes rather than broader clinical service and organisation influence. Formal output from master's research was low, but ongoing research engagement and activity was high suggesting master's degree training is an under-recognised source for AHP research capacity building. Future research should investigate the potential benefits of better coordinated and prioritised research at master's degree level on professional and organisational impact.

The decision maker's lament: If I only had some science!

Environmental decision makers lament instances in which the lack of actionable science limits confident decision-making. Their reaction when the needed scientific information is of poor quality, uninformative, unintelligible, or altogether absent is often to criticize scientists, their work, or science in general. The considerations offered here encourage decision makers to explore alternative approaches to alleviate their disappointment. Ironically, many researchers lament the lack of support for the science they wish to deliver and accuse decision makers of failing to realize the value of the scientific studies they propose. Both communities would benefit by remembering that producing actionable science for a pending decision requires knowing the context for that decision beforehand. They may also look inward. Only then will they find answers to the question: What can I do within my own capacity to ensure that the necessary actionable science becomes available and facilitate its use to inform decisions?

The societal impact of Open Science: a scoping review

Open Science (OS) aims, in part, to drive greater societal impact of academic research. Government, funder and institutional policies state that it should further democratize research and increase learning and awareness, evidence-based policy-making, the relevance of research to society's problems, and public trust in research. Yet, measuring the societal impact of OS has proven challenging and synthesized evidence of it is lacking. This study fills this gap by systematically scoping the existing evidence of societal impact driven by OS and its various aspects, including Citizen Science (CS), Open Access (OA), Open/FAIR Data (OFD), Open Code/Software and others. Using the PRISMA Extension for Scoping Reviews and searches conducted in Web of Science, Scopus and relevant grey literature, we identified 196 studies that contain evidence of societal impact. The majority concern CS, with some focused on OA, and only a few addressing other aspects. Key areas of impact found are education and awareness, climate and environment, and social engagement. We found no literature documenting evidence of the societal impact of OFD and limited evidence of societal impact in terms of policy, health, and trust in academic research. Our findings demonstrate a critical need for additional evidence and suggest practical and policy implications.

Multivariate mixed-effects ordinal logistic regression models with difference-in-differences estimator of the impact of WORTH Yetu on household hunger and socioeconomic status among OVC caregivers in Tanzania

BACKGROUND

Although most of the livelihood programmes target women, those that involve women and men have been evaluated as though men and women were a single homogenous population, with a mere inclusion of gender as an explanatory variable. This study evaluated the impact of WORTH Yetu (an economic empowerment intervention to improve livelihood outcomes) on household hunger, and household socioeconomic status (SES) among caregivers (both women and men) of orphaned and vulnerable children (OVC) in Tanzania. The study hypothesized that women and men respond to livelihood interventions differently, hence a need for gender-disaggregated impact evaluation of such interventions.

METHODS

This is a secondary analysis of longitudinal data, involving caregivers' baseline (2016-2019) and follow-up (2019-2020) data from the USAID Kizazi Kipya project in 25 regions of Tanzania. Two dependent variables (ie, outcomes) were assessed; household hunger which was measured using the Household Hunger Scale (HHS), and Socioeconomic Status (SES) using the Principal Component Analysis (PCA). WORTH Yetu, a livelihood intervention implemented by the USAID Kizazi Kipya project was the main independent variable whose impact on the two outcomes was evaluated using multivariate analysis with a multilevel mixed-effects, ordinal logistic regression model with difference-in-differences (DiD) estimator for impact estimation.

RESULTS

The analysis was based on 497,293 observations from 249,655 caregivers of OVC at baseline, and 247,638 of them at the follow-up survey. In both surveys, 70% were women and 30% were men. Their mean age was 49.3 (±14.5) years at baseline and 52.7 (±14.8) years at the follow-up survey. Caregivers' membership in WORTH Yetu was 10.1% at the follow-up. After adjusting for important confounders there was a significant decline in the severity of household hunger by 46.4% among WORTH Yetu members at the follow-up compared to the situation at the baseline (adjusted Odds Ratio (aOR) = 0.536, 95% Confidence Interval (CI) [0.521, 0.553]). The decline was 45.7% among women (aOR = 0.543 [0.524, 0.563]) and 47.5% among men (aOR = 0.525 [0.497, 0.556]). Regarding SES, WORTH Yetu members were 15.9% more likely to be in higher wealth quintiles at the follow-up compared to the situation at the baseline (aOR = 1.159 [1.128, 1.190]). This impact was 20.8% among women (aOR = 1.208 [1.170, 1.247]) and 4.6% among men (aOR = 1.046 [0.995, 1.101]).

CONCLUSION

WORTH Yetu was associated with a significant reduction in household hunger, and a significant increase in household SES among OVC caregivers in Tanzania within an average follow-up period of 1.6 years. The estimated impacts differed significantly by gender, suggesting that women and men responded to the WORTH Yetu intervention differently. This implied that the design, delivery, and evaluation of such programmes should happen in a gender responsive manner, recognising that women and men are not the same with respect to the programmes.

Key informants perspectives on creating a high impact research department in family and community medicine: a qualitative project

BACKGROUND

Primary care is integral to the health system and population health. Primary care research is still in development and most academic departments lack effective research investments. High impact primary care research programs are needed to advance the field to ensure a robust primary care system for the future. The project objective was to understand key informants' views of structures, functions, and processes required to create a high impact research program in an academic primary care department.

METHODS

A descriptive qualitative project with key informants from research programs in primary care. Participants included international research leaders in primary care (n = 10), department of family and community researchers (n = 37) and staff (n = 9) in an academic primary care department, other university leaders (n = 3) and members of the departmental executive leadership team (1 department; 25 members). Semi-structured interviews (n = 27), and focus groups (n = 6) were audio recorded, transcribed, and analyzed using thematic analysis. We used a socioecological framework which described micro, meso, macro levels of influence.

RESULTS

At the micro level despite barriers with respect to funding, protected time and lack of formal mentorship, personal motivation was a key factor. At the meso level, the organizational structure that promoted collaboration and a sense of connection emerged as a key factor. Specifically research leaders identified a research faculty development pipeline based on equity, diversity, inclusion, indigeneity, and accessibility principles with thematic areas of focus as key enablers. Lastly, at the macro level, an overarching culture and policies that promoted funding and primary care research was associated with high impact programs.

CONCLUSION

The alignment/complementarity of micro, meso, and macro level factors influenced the creation of a high impact research department in primary care. High impact research in primary care is facilitated by the development of researchers through formalized and structured mentorship/sponsorship and a department culture that promote primary care research.

Various application roles for Campbell systematic reviews: a citation analysis

OBJECTIVES

Systematic reviews (SRs) are becoming essential evidence in the decision-making process within the field of social sciences. This study aimed to investigate how Campbell SRs were cited and explore their specific application roles.

STUDY DESIGN AND SETTING

We included Campbell SRs published between 2016 and 2020 by searching the Wiley online library, and retrieved the articles and documents citing Campbell SRs from the Web of Science and Google Scholar by December 31, 2021. We described the characteristics of the SRs and citations, and formulated a set of application roles by analyzing the sentences or paragraphs where the SRs were cited.

RESULTS

Sixty nine Campbell SRs were published between 2016 and 2020; they were cited in 641 articles or documents a total of 1,289 times. The primary types of articles that cited Campbell SRs were cross-sectional studies (n = 226, 35.3%), SRs (n = 112, 17.5%), randomized controlled trials (n = 77, 12.0%), and policy reports (n = 57, 8.8%). Articles utilizing Campbell SRs were predominantly led by authors from the United States (n = 184, 28.7%), the United Kingdom (n = 98, 15.3%), and Australia (n = 51, 8.0%). We formulated a set of 10 application roles for the Campbell SRs, of which the most frequent were: describing the current status in the field of interest (n = 691, 53.6%), corroboration of the results (n = 140, 10.9%), identifying research gaps (n = 130, 10.1%), and providing methodological references (n = 126, 9.8%); the role of supporting policy recommendations or decisions accounted for 6.0% (n = 77) of the citations. Approximately 12% of Campbell SRs were used to support policy recommendations or decisions.

CONCLUSION

Campbell SRs are widely applied, particularly in scientific research, to describe the current status in the field of interest. Although the current application of Campbell SRs in supporting policy recommendations and decisions may not be predominant, there is a growing recognition of their value in using Campbell SRs to inform decision-making.

Prioritising and incentivising productivity within indicator-based approaches to Research Impact Assessment: a commentary

Research Impact Assessment (RIA) represents one of a suite of policies intended to improve the impact generated from investment in health and medical research (HMR). Positivist indicator-based approaches to RIA are widely implemented but increasingly criticised as theoretically problematic, unfair, and burdensome. This commentary proposes there are useful outcomes that emerge from the process of applying an indicator-based RIA framework, separate from those encapsulated in the metrics themselves. The aim for this commentary is to demonstrate how the act of conducting an indicator-based approach to RIA can serve to optimise the productive gains from the investment in HMR. Prior research found that the issues regarding RIA are less about the choice of indicators/metrics, and more about the discussions prompted and activities incentivised by the process. This insight provides an opportunity to utilise indicator-based methods to purposely optimise the research impact. An indicator-based RIA framework specifically designed to optimise research impacts should: focus on researchers and the research process, rather than institution-level measures; utilise a project level unit of analysis that provides control to researchers and supports collaboration and accountability; provide for prospective implementation of RIA and the prospective orientation of research; establish a line of sight to the ultimate anticipated beneficiaries and impacts; Include process metrics/indicators to acknowledge interim steps on the pathway to final impacts; integrate 'next' users and prioritise the utilisation of research outputs as a critical measure; Integrate and align the incentives for researchers/research projects arising from RIA, with those existing within the prevailing research system; integrate with existing peer-review processes; and, adopt a system-wide approach where incremental improvements in the probability of translation from individual research projects, yields higher impact across the whole funding portfolio.Optimisation of the impacts from HMR investment represents the primary purpose of Research Impact policy. The process of conducting an indicator-based approach to RIA, which engages the researcher during the inception and planning phase, can directly contribute to this goal through improvements in the probability that an individual project will generate interim impacts. The research project funding process represents a promising forum to integrate this approach within the existing research system.

Research Enterprise of the Congenital Heart Surgeons' Society 2018 to 2023: History, Accomplishments, Transformation, and Challenges

The history of the research arm of the Congenital Heart Surgeons' Society (CHSS) through 2017 was contained within two prior publications that covered CHSS history in general. The present article is the first to focus explicitly on the research enterprise, with emphasis on the period 2018 to 2023. During this time, the challenges of continuing to build multiple cohorts with lifelong follow-up and to transform the enterprise to a premiere research organization became manifest. Although continuing its commitment to produce impactful research results and to educate the Kirklin/Ashburn Fellow, the research team devoted considerable effort to defining the problems of cohort relevance, workflow, data management, long-term patient follow-up, CHSS member engagement, and the regulatory burden. The team wrote a proposal outlining ways to solve the challenges. A major change from a single-institution "Data Center" to a two-institution Center for Research & Quality (CRQ) was made, assuring increased faculty members and resources. The proposed changes to structure and process began execution in mid-2022. A second Kirklin/Ashburn Fellowship position was created. Between 2018 and 2022, the CRQ produced 17 publications and launched five new research initiatives. This article chronicles the exciting five-year period in which the CHSS research enterprise began a transformation with the intent to become the premiere research organization in the world in the specialty of congenital cardiac surgery.

The Academic Impact of Congenital Heart Surgeons' Society (CHSS) Studies

PURPOSE

We reviewed all 64 articles ever published by The Congenital Heart Surgeons' Society (CHSS) Data Center to estimate the academic impact of these peer-reviewed articles.

MATERIALS AND METHODS

The Congenital Heart Surgeons' Society has performed research based on 12 Diagnostic Inception Cohorts. The first cohort (Transposition) began enrolling patients on January 1, 1985. We queried PubMed to determine the number of publications that referenced each of the 64 journal articles generated by the datasets of the 12 Diagnostic Inception Cohorts that comprise the CHSS Database. Descriptive summaries of the data were tabulated using mean with standard deviation and median with range.

RESULTS

Sixty-four peer-reviewed papers have been published based on the CHSS Database. Fifty-nine peer-reviewed articles have been published based on the 12 Diagnostic Inception Cohorts, and five additional articles have been published based on Data Science. Excluding the recently established Diagnostic Inception Cohort for patients with Ebstein malformation of tricuspid valve, the number of papers published per cohort ranged from 1 for coarctation to 11 for transposition of the great arteries. The 11 articles generated from the CHSS Transposition Cohort were referenced by a total of 111 articles (median number of references per journal article = 9 [range = 0-22, mean = 10.1]). Overall, individual articles were cited by an average of 11 (mean), and a maximum of 41 PubMed-listed publications. Overall, these 64 peer-reviewed articles based on the CHSS Database were cited 692 times in PubMed-listed publications. The first CHSS peer-reviewed article was published in 1987, and during the 35 years from 1987 to 2022, inclusive, the annual number of CHSS publications has ranged from 0 to 7, with a mean of 1.8 publications per year (median = 1, mode = 1).

CONCLUSION

Congenital Heart Surgeons' Society studies are widely referenced in the pediatric cardiac surgical literature, with over 10 citations per published article. These cohorts provide unique information unavailable in other sources of data. A tool to access this analysis is available at: [https://data-center.chss.org/multimedia/files/2022/CAI.pdf].

Evidence-based research impact praxis: Integrating scholarship and practice to ensure research benefits society

Effective research impact development is essential to address global challenges. This commentary highlights key issues facing research impact development as a nascent professional field of practice. We argue that those working on research impact should take a strategic, 'evidence-based' approach to maximize potential research benefits and minimize potential harms. We identify key features of evidence-based good practice in the context of research impact work. This includes integrating relevant research and theory into professional decision-making, drawing on a diversity of academic disciplines offering pertinent insights. Such an integration of scholarship and practice will improve the capacity of research impact work to make a positive difference for society. Moving the focus of research impact work to earlier stages in the research and innovation process through stakeholder engagement and anticipatory research can also boost its effectiveness. The research impact evidence base should be combined with the right kind of professional capacities and practical experience to enhance positive impact. Such capacities need to be developed through relevant education and training, for example, in participatory methods and social inclusion. Such training for research impact work needs to forge strong links between research impact scholarship and practice. Finally, there is a need for improvements in the evidence base for research impact to make it more practically useful.

Interrogating Our Views on the Impact of Education-Related Scholarship

From junior faculty members to seasoned full professors, pharmacy educators have likely all felt pressure to focus on peer-reviewed publication. Although publication is an important part of an academician's work, have we missed something critical by not focusing greater attention on a more inclusive conceptualization of education-related scholarship's impact? How can we describe the full impact of our education-related scholarship beyond traditional metrics (ie, publications, presentations, and grant funding) if the issue is not critically examined? With the growing expectations for scholarly teaching and interest in the Scholarship of Teaching and Learning in academic pharmacy in both the United States and Canada, this commentary examines and questions the current, often narrow, views on pharmacy educators' scholarly impact. In addition, it provides a new definition of education-related impact to encourage a more expansive view.

Participating in innovative medicines initiative funded neurodegenerative disorder projects—An impact analysis conducted as part of the NEURONET project

The European Commission's Innovative Medicines Initiative (IMI) has funded many projects focusing on neurodegenerative disorders (ND) that aimed to improve the diagnosis, prevention, treatment and understanding of NDs. To facilitate collaboration across this project portfolio, the IMI funded the “NEURONET” project between March 2019 and August 2022 with the aim of connecting these projects and promoting synergies, enhancing the visibility of their findings, understanding the impact of the IMI funding and identifying research gaps that warrant more/new funding. The IMI ND portfolio currently includes 20 projects consisting of 270 partner organizations across 25 countries. The NEURONET project conducted an impact analysis to assess the scientific and socio-economic impact of the IMI ND portfolio. This was to better understand the perceived areas of impact from those directly involved in the projects. The impact analysis was conducted in two stages: an initial stage developed the scope of the project, defined the impact indicators and measures to be used. A second stage designed and administered the survey amongst partners from European Federation of Pharmaceutical Industries and Associations (EFPIA) organizations and other partners (hereafter, referred to as “non-EFPIA” organizations). Responses were analyzed according to areas of impact: organizational, economic, capacity building, collaborations and networking, individual, scientific, policy, patient, societal and public health impact. Involvement in the IMI ND projects led to organizational impact, and increased networking, collaboration and partnerships. The key perceived disadvantage to project participation was the administrative burden. These results were true for both EFPIA and non-EFPIA respondents. The impact for individual, policy, patients and public health was less clear with people reporting both high and low impact. Overall, there was broad alignment between EFPIA and non-EFPIA participants' responses apart from for awareness of project assets, as part of scientific impact, which appeared to be slightly higher among non-EFPIA respondents. These results identified clear areas of impact and those that require improvement. Areas to focus on include promoting asset awareness, establishing the impact of the IMI ND projects on research and development, ensuring meaningful patient involvement in these public-private partnership projects and reducing the administrative burden associated with participation in them.

Do funding sources complement or substitute? Examining the impact of cancer research publications

Academic research often draws on multiple funding sources. This paper investigates whether complementarity or substitutability emerges when different types of funding are used. Scholars have examined this phenomenon at the university and scientist levels, but not at the publication level. This gap is significant since acknowledgement sections in scientific papers indicate publications are often supported by multiple funding sources. To address this gap, we examine the extent to which different funding types are jointly used in publications, and to what extent certain combinations of funding are associated with higher academic impact (citation count). We focus on three types of funding accessed by UK-based researchers: national, international, and industry. The analysis builds on data extracted from all UK cancer-related publications in 2011, thus providing a 10-year citation window. Findings indicate that, although there is complementarity between national and international funding in terms of their co-occurrence (where these are acknowledged in the same publication), when we evaluate funding complementarity in relation to academic impact (we employ the supermodularity framework), we found no evidence of such a relationship. Rather, our results suggest substitutability between national and international funding. We also observe substitutability between international and industry funding.

What is the societal impact of university research? A policy-oriented review to map approaches, identify monitoring methods and success factors

The discussion on the 'societal impact of university research' has emerged within the recent decade with different utilisation from the British REF impact stories to the EU Framework Programmes' criteria. The fundamental idea is that (university) research should contribute to addressing societal challenges, such as climate change, and not just pursue 'scientific excellence'. Several approaches have been developed, from case studies of the 'productive interactions' that co-create knowledge with societal stakeholders to the so-called 'altmetrics' tracking online dissemination of research outputs. Nonetheless, these experiences do not always point in the same direction, and the notion of 'societal impact' seems still unclearly defined. Given the growing policy pressure on universities to address societal challenges, this paper reviews the scientific literature on the so-called 'societal impact' of university research. By querying three scientific archives (Web of Science, Scopus and ScienceDirect), 135 relevant scientific publications are selected and discussed, looking for conceptualisations, monitoring methods and success factors. The review maps a value-laden discussion arguing for longer-term, multi-dimensional perspectives on university research impacts beyond bibliometric indicators. Findings highlight a progressive shift from 'attribution', i.e., looking for causal relationships between research and societal changes, to 'contribution', acknowledging researchers' efforts to engage with societal challenges.

How can impact strategies be developed that better support universities to address twenty-first-century challenges?

To better address twenty-first-century challenges, research institutions often develop and publish research impact strategies, but as a tool, impact strategies are poorly understood. This study provides the first formal analysis of impact strategies from the UK, Canada, Australia, Denmark, New Zealand and Hong Kong, China, and from independent research institutes. Two types of strategy emerged. First, ‘achieving impact’ strategies tended to be bottom-up and co-productive, with a strong emphasis on partnerships and engagement, but they were more likely to target specific beneficiaries with structured implementation plans, use boundary organisations to co-produce research and impact, and recognise impact with less reliance on extrinsic incentives. Second, ‘enabling impact’ strategies were more top-down and incentive-driven, developed to build impact capacity and culture across an institution, faculty or centre, with a strong focus on partnerships and engagement, and they invested in dedicated impact teams and academic impact roles, supported by extrinsic incentives including promotion criteria. This typology offers a new way to categorise, analyse and understand research impact strategies, alongside insights that may be used by practitioners to guide the design of future strategies, considering the limitations of top-down, incentive-driven approaches versus more bottom-up, co-productive approaches.

Digital artifacts reveal development and diffusion of climate research

Research for development organizations generate tremendous amount of accessible knowledge, but given their scale, time and resource constraints, the impact of outputs is not systematically analyzed. This is because traditional bibliometric analyses present limitations to synthesize accumulated knowledge and retrofitting indicators to historical outputs. To address these shortcomings, this study proposes an integrated, web-based approach to systematically analyze the production and diffusion of knowledge from large-scale research programs, using climate research of the International Maize and Wheat Improvement Center (CIMMYT) as a case study. Our analytical framework employs text mining, social network analysis and hyperlink analysis to an unstructured mass of publicly available digital artifacts such as institutional repositories, citation databases, and social media to uncover narratives, dynamics, and relationships. Findings show CIMMYT's climate research is strongly incorporated into a holistic systems approach and that the institution is actively engaged in knowledge exchanges with key actors from the scientific, development and public policy communities. The proposed analytical framework establishes an effective approach for research for development organizations to leverage existing online data sources to assess the extent of their knowledge production, dissemination, and reach.

What funders are doing to assess the impact of their investments in health and biomedical research

As pressures to maximize research funding grow, biomedical research funders are increasingly tasked with demonstrating the long-term and real-world impacts of their funded research investments. Over the past three decades, research impact assessments (RIA) have emerged as an important tool for analysing the impacts of research by incorporating logic models, frameworks and indicators to track measures of knowledge production, capacity-building, development of research products, adoption of research into clinical guidelines and policies, and the realization of health, economic and social benefits. While there are currently several models for RIA within the literature, less attention has been paid to how funders can practically select and implement a RIA model to demonstrate the impacts of their own research portfolios. In this paper, a literature review was performed to understand (1) which research funders have performed RIAs of their research portfolios to date; (2) how funders have designed their assessments, including the models and tools they have used; (3) what challenges to and facilitators of success have funders found when adopting the RIA model to their own portfolio; and (4) who participates in the assessments. Forty-four papers from both published and grey literature were found to meet the review criteria and were examined in detail. There is a growing culture of RIA among funders, and included papers spanned a diverse set of funders from 10 countries or regions. Over half of funders (59.1%) used a framework to conduct their assessment, and a variety of methods for collecting impact data were reported. Issues of methodological rigour were observed across studies in the review, and this was related to numerous challenges funders faced in designing timely RIAs with quality impact data. Over a third of articles (36.4%) included input from stakeholders, yet only one article reported surveying patients and members of the public as part of the assessment. To advance RIA among funders, we offer several recommendations for increasing the methodological rigour of RIAs and suggestions for future research, and call for a careful reflection of the voices needed in an impact assessment to ensure that RIAs are having a meaningful impact on patients and the public.

Lessons from bright-spots for advancing knowledge exchange at the interface of marine science and policy

Evidence-informed decision-making is in increasing demand given growing pressures on marine environments. A way to facilitate this is by knowledge exchange among marine scientists and decision-makers. While many barriers are reported in the literature, there are also examples whereby research has successfully informed marine decision-making (i.e., 'bright-spots'). Here, we identify and analyze 25 bright-spots from a wide range of marine fields, contexts, and locations to provide insights into how to improve knowledge exchange at the interface of marine science and policy. Through qualitative surveys we investigate what initiated the bright-spots, their goals, and approaches to knowledge exchange. We also seek to identify what outcomes/impacts have been achieved, the enablers of success, and what lessons can be learnt to guide future knowledge exchange efforts. Results show that a diversity of approaches were used for knowledge exchange, from consultative engagement to genuine knowledge co-production. We show that diverse successes at the interface of marine science and policy are achievable and include impacts on policy, people, and governance. Such successes were enabled by factors related to the actors, processes, support, context, and timing. For example, the importance of involving diverse actors and managing positive relationships is a key lesson for success. However, enabling routine success will require: 1) transforming the ways in which we train scientists to include a greater focus on interpersonal skills, 2) institutionalizing and supporting knowledge exchange activities in organizational agendas, 3) conceptualizing and implementing broader research impact metrics, and 4) transforming funding mechanisms to focus on need-based interventions, impact planning, and an acknowledgement of the required time and effort that underpin knowledge exchange activities.

The Development of Green Bond in Developing Countries: Insights from Southeast Asia Market Participants

Given the increasing importance of green bond as the main funding source for the Sustainable Development Goals, the green bond is an emerging concept in the region of Southeast Asia. In addition, the concurrent Covid-19 pandemic has caused disruption to the development of green bond around the world. This research explores the current development status of the green bond in Southeast Asian countries. A total of thirty-two semi-structured interviews were held with capital market participants in Southeast Asian countries. The results highlight barriers, opportunities, and regulation difficulties, and expected growth for the development of the green bond market. This research is concluded by indicating several propositions that can be tested in the future to generalize the findings from this work. We thus extend the knowledge of green bond in the financial markets of Southeast Asian countries, which also delivers implications for practitioners and policy-makers regarding the development of green bond in Southeast Asian countries.

SDGs: A Responsible Research Assessment Tool toward Impactful Business Research

An alternative research assessment (RA) tool was constructed to assess the relatedness of published business school research to the United Nations’ 17 Sustainable Development Goals (SDGs). The RA tool was created using Leximancer™, an on-line cloud-based text analytic software tool, that identified core themes within the SDG framework. Eight (8) core themes were found to define the ‘spirit of the SDGs’: Sustainable Development, Governance, Vulnerable Populations, Water, Gross Domestic Product (GDP), Food Security, Restoration, and Public Health. These themes were compared to the core themes found in the content of 4576 academic articles published in 2019 in journals that comprise the Financial Times (FT) 50 list. The articles’ relatedness to the SDG themes were assessed. Overall, 10.6% of the themes found in the FT50 journal articles had an explicit relationship to the SDG themes while 24.5% were implied. Themes generated from machine learning (ML), augmented by researcher judgement (to account for synonyms, similar concepts, and discipline specific examples), improved the robustness of the relationships found between the SDG framework and the published articles. Although there are compelling reasons for business schools to focus research on advancing the SDGs, this study and others highlight that there is much opportunity for improvement. Recommendations are made to better align academic research with the SDGs, influencing how business school faculty and their schools prioritize research and its role in the world.

Impacts of clinical academic activity: qualitative interviews with healthcare managers and research-active nurses, midwives, allied health professionals and pharmacists

OBJECTIVES

To explore the perceived impacts of clinical academic activity among the professions outside medicine.

DESIGN

Qualitative semistructured interviews.

SETTING AND PARTICIPANTS

There were two groups of interviewees: Research-active nurses, midwives, allied health professionals, healthcare scientists, psychologists and pharmacists (NMAHPPs) and managers of these professions. All participants were employed in a single, multisite healthcare organisation in the UK.

ANALYSIS

Interview transcripts were analysed using the framework method to identify key themes, subthemes and areas of divergence.

RESULTS

Four themes were identified. The first, cultural shifts, described the perceived improvements in the approach to patient care and research culture that were associated with clinical academic activity. The second theme explored visibility and included the positive reputation that clinical academics were identified as bringing to the organisation in contrast with perceived levels of invisibility and inaccessibility of these roles. The third theme identified the impacts of the clinical academic pathways, including the precarity of these roles. The final theme explored making impact tangible, and described interviewees' suggestions of possible methods to record and demonstrate impact.

CONCLUSIONS

Perceived positive impacts of NMAHPP clinical academic activity focused on interlinked positive changes for patients and clinical teams. This included delivery of evidence-based healthcare, patient involvement in clinical decision making and improved staff recruitment and retention. However, the positive impacts of clinical academic activity often centred around individual clinicians and did not necessarily translate throughout the organisation. The current clinical academic pathway was identified as causing tension between the perceived value of clinical academic activity and the need to find sufficient staffing to cover clinical services.

A qualitative systematic review and thematic synthesis exploring the impacts of clinical academic activity by healthcare professionals outside medicine

BACKGROUND

There are increasing opportunities for healthcare professionals outside medicine to be involved in and lead clinical research. However, there are few roles within these professions that include time for research. In order to develop such roles, and evaluate effective use of this time, the range of impacts of this clinical academic activity need to be valued and understood by healthcare leaders and managers. To date, these impacts have not been comprehensively explored, but are suggested to extend beyond traditional quantitative impact metrics, such as publications, citations and funding awards.

METHODS

Ten databases, four grey literature repositories and a naïve web search engine were systematically searched for articles reporting impacts of clinical academic activity by healthcare professionals outside medicine. Specifically, this did not include the direct impacts of the research findings, rather the impacts of the research activity. All stages of the review were performed by a minimum of two reviewers and reported impacts were categorised qualitatively according to a modified VICTOR (making Visible the ImpaCT Of Research) framework.

RESULTS

Of the initial 2704 identified articles, 20 were eligible for inclusion. Identified impacts were mapped to seven themes: impacts for patients; impacts for the service provision and workforce; impacts to research profile, culture and capacity; economic impacts; impacts on staff recruitment and retention; impacts to knowledge exchange; and impacts to the clinical academic.

CONCLUSIONS

Several overlapping sub-themes were identified across the main themes. These included the challenges and benefits of balancing clinical and academic roles, the creation and implementation of new evidence, and the development of collaborations and networks. These may be key areas for organisations to explore when looking to support and increase academic activity among healthcare professionals outside medicine. The modified VICTOR tool is a useful starting point for individuals and organisations to record the impact of their research activity. Further work is needed to explore standardised methods of capturing research impact that address the full range of impacts identified in this systematic review and are specific to the context of clinical academics outside medicine.