Implementing Common Metrics across the NIH Clinical and Translational Science Awards (CTSA) consortium

Implementing a mentoring program for clinical research professionals: A novel professional development initiative for university health research staff

Clinical and translational research relies on a well-trained workforce, but mentorship programs designed expressly for this workforce are lacking. This paper presents the development of a mentoring program for research staff and identifies key programmatic outcomes. Research staff participating in this program were matched with a senior mentor. Focus groups were conducted to identify key program outcomes. Surveys were administered throughout the program period to assess participants' experience, gains in skill, and subsequent careers. Analysis of the resultant qualitative and quantitative data are used to characterize the implementation and impact of the program. A total of 47 mentees and 30 mentors participated in program between 2018 and 2023. A comprehensive logic model of short-, intermediate- and long-term outcomes was developed. Participants reported positive valuations of every programmatic outcome assessed including their program experience, learning and research careers. The pool of available mentors also grew as new mentors were successfully recruited for each cohort. This mentorship program developed and implemented by senior research staff successfully provided junior research staff with professional development support, mentorship, and professional development opportunities. Junior and senior health research staff built mentoring relationships that advanced their clinical and translational research careers.

From insights to action: Enriching the clinical and translational research workforce by developing diverse and inclusive career programs

Diversification of the Translational Science workforce is a strategic goal for the National Center for the Advancement of Translational Science (NCATS) program. NCATS has identified the development of translational science education, training, and support for a diverse translational science workforce as key to advancing the growing field of translational science. An annual mixed-methods assessment has been conducted on Common Metrics data submitted by over 60 Clinical & Translational Science Awards (CTSA) programs nationwide and includes metrics addressing recruitment and retention of scientists with particular attention to underrepresented persons and women. This article describes a methodology for the development of From Insights to Action, a resource for guiding program implementation and strategic planning to develop a diverse clinical and translational science workforce. This was informed by the Common Metrics Initiative process and constituted of findings from qualitative interviews of a subset of CTSAs that participated. The dissemination of this guide had several impacts, including providing structural foci for the CTSA Fall 2020 program meeting centered on Diversity, Equity, and Inclusion in translational science; addressing NCATS' goal of workforce diversity; and understanding the number of diverse graduates still engaged in research.

Design and implementation of an integrated data model to support clinical and translational research administration

OBJECTIVE

Both academic medical centers and biomedical research sponsors need to understand impact of scientific funding to determine value. For the National Institutes of Health (NIH) Clinical and Translational Science Award (CTSA) hubs, tracking research activities can be complex, often involving multiple institutions and continually changing federal reporting requirements. Existing research administrative systems are institution-specific and tend to focus only on parts of a greater whole. The goal of this case report is to describe a comprehensive data model that addresses this gap.

MATERIALS AND METHODS

Web-based Center Administrative Management Program (WebCAMP) has been developed over a period of over 15 years in the context of CTSA hubs, with the recent addition of T32 programs. Its data model centers around the key concepts of people, projects, resources (inputs), and outcomes (outputs).

RESULTS

The WebCAMP data model and associated toolset for biomedical research administration integrates multiple components of the research enterprise, has been used by our CTSA hub for over 15 years and has been adopted by more than 20 other CTSA hubs.

DISCUSSION

To the best of our knowledge, this study is among the first to describe a comprehensive data model for biomedical research administration. Opportunities for future work include improved grant tracking through the development of a universal identifier that spans public and private funders, and a more generic outcomes tracking model able to rapidly incorporate new outcome types.

CONCLUSION

We propose that the WebCAMP data model, or a derivative of it, could serve as a future standard for research administrative data warehousing.

Moving beyond annual data reports: A blueprint for communicating and disseminating actionable intelligence

Identifying and disseminating actionable intelligence is a challenging task that requires thoughtful planning. The National Center for Advancing Translational Sciences instituted the Common Metrics Initiative with the goal of evaluating the Clinical and Translational Science Awards (CTSA) Programs using a standard set of metrics. Initially managed by Tufts University, the Center for Leading Innovation and Collaboration (CLIC) at the University of Rochester began leading this initiative in 2017. In directing this work, CLIC created a framework for communicating and disseminating data insights. Insights to Inspire emerged from the need to share strategies and lessons learned to improve metric performance at the local level to a network of 60+ academic research institutions. Insights to Inspire employs a mixed methods approach for translating data into actionable intelligence. A series of blogs, webinars, and webcasts were designed to communicate metric-specific strategies used by individual sites to the broader CTSA consortium. A dissemination plan to expand the reach beyond metric stakeholders utilized focused communications including social media channels, network newsletters, and presentations at national meetings. This framework serves as a blueprint for other national evaluation programs interested in a systematic approach to using data insights for continuous improvement.

Feasibility and acceptability of a structured quality by design approach to enhancing the rigor of clinical studies at an academic health center

Introduction:

Clinical trials are a critical step in the meaningful translation of biomedical discoveries into effective diagnostic and therapeutic interventions. Quality by design (QbD) is a framework for embedding quality into the design, conduct, and monitoring of clinical trials. Here we report the feasibility and acceptability of a process for implementing QbD in clinical research at an academic health center via multidisciplinary design studios aimed at identifying and prioritizing critical to quality (CTQ) factors.

Methods:

The Clinical Trial Transformation Initiative’s Principles Document served as a guide to identify and categorize key CTQ factors, defined as elements of a clinical trial that are critical to patient safety and data integrity. Individual trials were reviewed in CTQ design studios (CTQ-DS) and the feasibility and acceptability of this intervention was examined through post-meeting interviews and surveys.

Results:

Eight clinical research protocols underwent the QbD evaluation process. The protocols ranged from multicenter randomized clinical trials to nonrandomized investigator-initiated studies. A developmental evaluation informed the iterative refinement of the CTQ-DS process, and post-meeting surveys revealed that CTQ-DS were highly valued by principal investigators (PIs) and resulted in multiple protocol changes.

Conclusions:

The present study demonstrated that QbD principles can be implemented to inform the design and conduct of clinical research at an academic health center using multidisciplinary design studios aimed at identifying and prioritizing CTQ elements. This approach was well received by the participants including study PIs. Future research will need to evaluate the effectiveness of this approach in improving the quality of clinical research.

Improving Translational Paradigms in Drug Discovery and Development

Despite improved knowledge regarding disease causality, new drug targets, and enabling technologies, the attrition rate for compounds entering clinical trials has remained consistently high for several decades, with an average 90% failure rate. These failures are manifested in an inability to reproduce efficacy findings from animal models in humans and/or the occurrence of unexpected safety issues, and reflect failures in T1 translation. Similarly, an inability to sequentially demonstrate compound efficacy and safety in Phase IIa, IIb, and III clinical trials represents failures in T2 translation. Accordingly, T1 and T2 translation are colloquially termed 'valleys of death'. Since T2 translation dealt almost exclusively with clinical trials, T3 and T4 translational steps were added, with the former focused on facilitating interactions between laboratory- and population-based research and the latter on 'real world' health outcomes. Factors that potentially lead to T1/T2 compound attrition include: the absence of biomarkers to allow compound effects to be consistently tracked through development; a lack of integration/'de-siloing' of the diverse discipline-based and technical skill sets involved in drug discovery; the industrialization of drug discovery, which via volume-based goals often results in quantity being prioritized over quality; inadequate project governance and strategic oversight; and flawed decision making based on unreliable/irreproducible or incomplete data. A variety of initiatives have addressed this problem, including the NIH National Center for Advancing Translational Sciences (NCATS), which has focused on bringing an unbiased academic perspective to translation, to potentially revitalize the process. This commentary provides an overview of the basic concepts involved in translation, along with suggested changes in the conduct of biomedical research to avoid valleys of death, including the use of Translational Scoring as a tool to avoid translational attrition and the impact of the FDA Accelerated Approval Pathway in lowering the hurdle for drug approval. © 2021 Wiley Periodicals LLC.

Do common metrics add value? Perspectives from NIH Clinical and Translational Science Awards (CTSA) Consortium hubs

INTRODUCTION

The Clinical and Translational Science Awards (CTSA) Consortium, a network of academic health care institutions with CTSA hubs, is charged with improving the national clinical and translational research enterprise. The CTSA Consortium and the NIH National Center for Advancing Translational Sciences implemented the Common Metrics Initiative comprised of standardized metrics and a shared performance improvement framework. This article summarizes hubs' perspectives on its value during the initial implementation.

METHODS

The value was assessed across 58 hubs. Survey items assessed change in perceived ability to manage performance and advance clinical and translational science. Semi-structured interviews elicited hubs' perspectives on meaningfulness and value-added of the Common Metrics Initiative and hubs' recommendations.

RESULTS

Hubs considered their abilities to manage performance to have improved, but there was no change in perceived ability to advance clinical and translational science. The initiative added value by providing a formal structured process, enabling strategic conversations, facilitating improvements in processes, providing an external impetus for improvement, and providing justification for funds invested. Hubs were concerned about the usefulness of the metrics chosen and whether the value-added was sufficient relative to the effort required. Hubs recommended useful benchmarking, disseminating best practices and promoting peer-to-peer learning, and expanding the use of data to inform the initiative.

CONCLUSIONS

Implementing Common Metrics and a performance improvement framework yielded concrete short-term benefits, but concerns about usefulness remained, particularly considering the effort required. The Common Metrics Initiative should focus on facilitating cross-hub collaboration around metrics that address high-priority impact areas for individual hubs and the Consortium.

Pilot test of an accrual Common Metric for the NIH Clinical and Translational Science Awards (CTSA) Consortium: Metric usefulness

The Common Metrics Initiative aims to develop and field metrics to improve research processes within the national Clinical and Translational Science Award (CTSA) Consortium. A Median Accrual Ratio (MAR) common metric was developed to assess the results of efforts to increase subject accrual into a set of clinical trials within the expected time period. A pilot test of the MAR was undertaken at Tufts Clinical and Translational Science Institute (CTSI) with eight CTSA Consortium hubs. Post-pilot interviews were conducted with 9 CTSA Principal Investigators (PIs) and 23 pilot team members. Over three-quarters (78%) of respondents reported that the MAR could be useful for performance improvement, but also described limitations or concerns. The most commonly cited barrier to MAR use for performance improvement was difficulty in interpreting the single value that is produced. Most respondents were interested in using the MAR to assess recruitment at an individual trial level. Majority of respondents (63%) had mixed opinions about aggregating metric results across the CTSA Consortium for comparison or benchmarking. Collecting data about additional contextual factors, and comparing accrual between subgroups, were cited as potentially helping address concerns about aggregation. Significant challenges remain in ensuring that the MAR can be sufficiently useful for collaborative process improvement. We offer recommendations to potentially improve metric usefulness.

Pilot test of an accrual Common Metric for the NIH Clinical and Translational Science Awards (CTSA) Consortium: Metric feasibility and data quality

Failure to accrue participants into clinical trials incurs economic costs, wastes resources, jeopardizes answering research questions meaningfully, and delays translating research discoveries into improved health. This paper reports the results of a pilot test of the Median Accrual Ratio (MAR) metric developed as a part of the Common Metrics Initiative of the NIH's National Center for Advancing Translational Science (NCATS) Clinical and Translational Science Award (CTSA) Consortium. Using the metric is intended to enhance the ability of the CTSA Consortium and its "hubs" to increase subject accrual into trials within expected timeframes. The pilot test was undertaken at Tufts Clinical and Translational Science Institute (CTSI) with eight CTSA Consortium hubs. We describe the pilot test methods, and results regarding feasibility of collecting metric data and the quality of data that was collected. Participating hubs welcomed the opportunity to assess accrual efforts, but experienced challenges in collecting accrual metric data due to insufficient infrastructure and inconsistent implementation of electronic data systems and lack of uniform data definitions. Also, the metric could not be constructed for all trial designs, particularly those using competitive enrollment strategies. We offer recommendations to address the identified challenges to facilitate progress to broad accrual metric data collection and use.

Evaluation of initial progress to implement Common Metrics across the NIH Clinical and Translational Science Awards (CTSA) Consortium

Introduction

The Clinical and Translational Science Awards (CTSA) Consortium, about 60 National Institutes of Health (NIH)-supported CTSA hubs at academic health care institutions nationwide, is charged with improving the clinical and translational research enterprise. Together with the NIH National Center for Advancing Translational Sciences (NCATS), the Consortium implemented Common Metrics and a shared performance improvement framework.

Methods

Initial implementation across hubs was assessed using quantitative and qualitative methods over a 19-month period. The primary outcome was implementation of three Common Metrics and the performance improvement framework. Challenges and facilitators were elicited.

Results

Among 59 hubs with data, all began implementing Common Metrics, but about one-third had completed all activities for three metrics within the study period. The vast majority of hubs computed metric results and undertook activities to understand performance. Differences in completion appeared in developing and carrying out performance improvement plans. Seven key factors affected progress: hub size and resources, hub prior experience with performance management, alignment of local context with needs of the Common Metrics implementation, hub authority in the local institutional structure, hub engagement (including CTSA Principal Investigator involvement), stakeholder engagement, and attending training and coaching.

Conclusions

Implementing Common Metrics and performance improvement in a large network of research-focused organizations proved feasible but required substantial time and resources. Considerable heterogeneity across hubs in data systems, existing processes and personnel, organizational structures, and local priorities of home institutions created disparate experiences across hubs. Future metric-based performance management initiatives across heterogeneous local contexts should anticipate and account for these types of differences.