Expanding assessments of translational research programs: supplementing metrics with value judgments

Development of a novel and more holistic approach for assessing impact in health and medical research: the Research Impact Assessment Framework

Considered investment in health and medical research (HMR) is critical for fostering a healthcare system that is sustainable, effective, responsive, and innovative. While several tools exist to measure the impact of research, few assess the research environment that nurtures and supports impactful research and the strategic alignment of research with societal needs. This perspective article discusses the limitations of existing assessment tools and presents a novel Research Impact Assessment Framework designed to enable more strategic and targeted investment towards HMR, having the potential for significant public benefit.

Design of a Multidisciplinary Training Program in Complementary and Integrative Health Clinical Research: Building Research Across Interdisciplinary Gaps

Despite wide use by the public, limited evidence is available for many complementary and integrative health (CIH) practices. Thus, clinical researchers knowledgeable about CIH disciplines are necessary to study the efficacy and effectiveness of CIH practices to benefit the public health. To partially address the need for clinical researchers versed in CIH, the authors of this study report the design of an interprofessional clinical research training program focused on CIH, the Building Research across Interdisciplinary Gaps (BRIDG) program, supported by a 5-year T90/R90 grant from the National Center for Complementary and Integrative Health. The T90-supported arm of the program trains doctoral-level CIH providers in clinical research at the research-intensive University of Washington. The R90-supported arm of the program trains researchers with conventional backgrounds in the practices of CIH at the clinic-intensive National University of Natural Medicine. The "Translational Science Spectrum" provides a common conceptual framework for both programs. Specific program elements include: individualized didactic training in clinical research and CIH disciplines; placement with clinical research mentors; placement with clinical mentors in CIH disciplines; shared and independent research project development; and interdisciplinary experiences through seminars and retreats. Program evaluation includes annual completion of the Clinical Research Appraisal Inventory (CRAI), which queries confidence in research skills and methods and periodic evaluation of training elements using the Supplemental Kellogg Logic-World Health Organization model, which emphasizes relevance, adequacy, efficiency, effectiveness, process, impact, equity, and sustainability. The BRIDG program exemplifies a new standard in interprofessional clinical research training, made possible through strong collaboration between disparate research- and clinically intensive institutions.

Trust in Community-Engaged Research Partnerships: A Methodological Overview of Designing a Multisite Clinical and Translational Science Awards (CTSA) Initiative

Community-engaged research (CEnR) builds on the strengths of the Clinical and Translational Science Awards (CTSA) framework to address health in underserved and minority communities. There is a paucity of studies that identify the process from which trust develops in CEnR partnerships. This study responds to the need for empirical investigation of building and maintaining trust from a multistakeholder perspective. We conducted a multi-institutional pilot study using concept mapping with to better understand how trust, a critical outcome of CEnR partnerships, can act as "social capital." Concept mapping was used to collect data from the three stakeholder groups: community, health-care, and academic research partners across three CTSAs. Concept mapping is a mixed-methods approach that allows participants to brainstorm and identify factors that contribute to a concept and describe ways in which those factors relate to each other. This study offers important insights on developing an initial set of trust measures that can be used across CTSAs to understand differences and similarities in conceptualization of trust among key stakeholder groups, track changes in public trust in research, identify both positive and negative aspects of trust, identify characteristics that maintain trust, and inform the direction for future research.

The Current Trend of the Translational Research Paradigm

The translational research paradigm is a process of discovering basic science concepts and applying the knowledge in clinical practice, aiming to improve patient care. The stages involved in the paradigm form a complex network of shared knowledge amongst research collaborators, including patients. This nature of the paradigm allows those involved to work together effectively. However, the translational research paradigm is often overlooked by many scientists, educators, and research institutions. Hence, a large amount of comprehensive and hugely invested research projects fail to make a scientific impact. We aim to outline and describe this paradigm in order to aid in the successful translation of effective research.

Measuring research impact in Australia's medical research institutes: a scoping literature review of the objectives for and an assessment of the capabilities of research impact assessment frameworks

BACKGROUND

Realising the economic potential of research institutions, including medical research institutes, represents a policy imperative for many Organisation for Economic Co-operation and Development nations. The assessment of research impact has consequently drawn increasing attention. Research impact assessment frameworks (RIAFs) provide a structure to assess research translation, but minimal research has examined whether alternative RIAFs realise the intended policy outcomes. This paper examines the objectives presented for RIAFs in light of economic imperatives to justify ongoing support for health and medical research investment, leverage productivity via commercialisation and outcome-efficiency gains in health systems, and ensure that translation and impact considerations are embedded into the research process. This paper sought to list the stated objectives for RIAFs, to identify existing frameworks and to evaluate whether the identified frameworks possessed the capabilities necessary to address the specified objectives.

METHODS

A scoping review of the literature to identify objectives specified for RIAFs, inform upon descriptive criteria for each objective and identify existing RIAFs. Criteria were derived for each objective. The capability for the existing RIAFs to realise the alternative objectives was evaluated based upon these criteria.

RESULTS

The collated objectives for RIAFs included accountability (top-down), transparency/accountability (bottom-up), advocacy, steering, value for money, management/learning and feedback/allocation, prospective orientation, and speed of translation. Of the 25 RIAFs identified, most satisfied objectives such as accountability and advocacy, which are largely sufficient for the first economic imperative to justify research investment. The frameworks primarily designed to optimise the speed of translation or enable the prospective orientation of research possessed qualities most likely to optimise the productive outcomes from research. However, the results show that few frameworks met the criteria for these objectives.

CONCLUSION

It is imperative that the objective(s) for an assessment framework are explicit and that RIAFs are designed to realise these objectives. If the objectives include the capability to pro-actively drive productive research impacts, the potential for prospective orientation and a focus upon the speed of translation merits prioritisation. Frameworks designed to optimise research translation and impact, rather than simply assess impact, offer greater promise to contribute to the economic imperatives compelling their implementation.

Feasibility of common bibliometrics in evaluating translational science

Introduction

A pilot study by 6 Clinical and Translational Science Awards (CTSAs) explored how bibliometrics can be used to assess research influence.

Methods

Evaluators from 6 institutions shared data on publications (4202 total) they supported, and conducted a combined analysis with state-of-the-art tools. This paper presents selected results based on the tools from 2 widely used vendors for bibliometrics: Thomson Reuters and Elsevier.

Results

Both vendors located a high percentage of publications within their proprietary databases (>90%) and provided similar but not equivalent bibliometrics for estimating productivity (number of publications) and influence (citation rates, percentage of papers in the top 10% of citations, observed citations relative to expected citations). A recently available bibliometric from the National Institutes of Health Office of Portfolio Analysis, examined after the initial analysis, showed tremendous potential for use in the CTSA context.

Conclusion

Despite challenges in making cross-CTSA comparisons, bibliometrics can enhance our understanding of the value of CTSA-supported clinical and translational research.

Multi-Dimensional Impact of the Public-Private Center for Translational Molecular Medicine (CTMM) in the Netherlands: Understanding New 21(st) Century Institutional Designs to Support Innovation-in-Society

Knowledge translation is at the epicenter of 21st century life sciences and integrative biology. Several innovative institutional designs have been formulated to cultivate knowledge translation. One of these organizational innovations has been the Center for Translational Molecular Medicine (CTMM), a multi-million public-private partnership in the Netherlands. The CTMM aims to accelerate molecular diagnostics and imaging technologies to forecast disease susceptibilities in healthy populations and early diagnosis and personalized treatment of patients. This research evaluated CTMM's impact on scientific, translational, clinical, and economic dimensions. A pragmatic, operationally-defined process indicators approach was used. Data were gathered from CTMM administrations, through a CTMM-wide survey (n = 167) and group interviews. We found that the CTMM focused on disease areas with high human, clinical, and economic burden to society (i.e., oncology, cardiovascular, neurologic, infection, and immunity diseases). CTMM displayed a robust scientific impact that rests 15%-80% above international reference values regarding publication volume and impact. Technology translation to the clinic was accelerated, with >50% of projects progressing from pre-clinical development to clinical testing within 5 years. Furthermore, CTMM has generated nearly 1500 Full Time Equivalent (FTE) of translational R&D capacity. Its positive impact on translational, (future) clinical, and economic aspects is recognized across all surveyed stakeholders. As organizational innovation is increasingly considered critical to forge linkages between life sciences discoveries and innovation-in-society, lessons learned from this study may inform other institutions with similar objectives such as the Clinical and Translational Science Awards (CTSA) Program of the National Institutes of Health (NIH) in the United States.

Evolution of Multidisciplinary Translational Teams (MTTs): Insights for Accelerating Translational Innovations

There is growing consensus about the factors critical for development and productivity of multidisciplinary teams, but few studies have evaluated their longitudinal changes. We present a longitudinal study of 10 multidisciplinary translational teams (MTTs), based on team process and outcome measures, evaluated before and after 3 years of CTSA collaboration. Using a mixed methods approach, an expert panel of five judges (familiar with the progress of the teams) independently rated team performance based on four process and four outcome measures, and achieved a rating consensus. Although all teams made progress in translational domains, other process and outcome measures were highly variable. The trajectory profiles identified four categories of team performance. Objective bibliometric analysis of CTSA-supported MTTs with positive growth in process scores showed that these teams tended to have enhanced scientific outcomes and published in new scientific domains, indicating the conduct of innovative science. Case exemplars revealed that MTTs that experienced growth in both process and outcome evaluative criteria also experienced greater innovation, defined as publications in different areas of science. Of the eight evaluative criteria, leadership-related behaviors were the most resistant to the interventions introduced. Well-managed MTTs demonstrate objective productivity and facilitate innovation.

Social network analysis to assess the impact of the CTSA on biomedical research grant collaboration

Success of the Clinical Translational Science Award (CTSA) program implicitly demands team science efforts and well-orchestrated collaboration across the translational silos (T1-T4). Networks have proven to be useful abstractions of research collaborations. Networks provide novel system-level insights and exhibit marked changes in response to external interventions, making them potential evaluation tools that complement more traditional approaches. This study is part of our ongoing efforts to assess the impact of the CTSA on Biomedical Research Grant Collaboration (BRGC). Collaborative research grants are a complex undertaking and an outcome of sustained interaction among researchers. In this report, BRGC networks representing collaborations among CTSA-affiliated investigators constructed from grants management system data at the University of Kentucky across a period of six years (2007-2012) corresponding to pre- and post-CTSA are investigated. Overlapping community structure detection algorithms, in conjunction with surrogate testing, revealed the presence of intricate research communities rejecting random graphs as generative mechanisms. The deviation from randomness was especially pronounced post-CTSA, reflecting an increasing trend in collaborations and team-science efforts potentially as a result of CTSA. Intercommunity cross talk was especially pronounced post-CTSA.