A "three-plus-one" evaluation model for clinical research management

Fundamental challenges in assessing the impact of research infrastructure

Clinical research infrastructure is one of the unsung heroes of the scientific response to the current COVID-19 pandemic. The extensive, long-term funding into research support structures, skilled people, and technology allowed the United Kingdom research response to move off the starting blocks at pace by utilizing pre-existing platforms. The increasing focus from funders on evaluating the outcomes and impact of research infrastructure investment requires both a reframing and progression of the current models in order to address the contribution of the underlying support infrastructure. The majority of current evaluation/outcome models focus on a "pipeline" approach using a methodology which follows the traditional research funding route with the addition of quantitative metrics. These models fail to embrace the complexity caused by the interplay of previous investment, the coalescing of project outputs from different funders, the underlying infrastructure investment, and the parallel development across different parts of the system. Research infrastructure is the underpinning foundation of a project-driven research system and requires long-term, sustained funding and capital investment to maintain scientific and technological expertise. Therefore, the short-term focus on quantitative metrics that are easy to collect and interpret and that can be assessed in a roughly 5-year funding cycle needs to be addressed. The significant level of investment in research infrastructure necessitates investment to develop bespoke methodologies that develop fit-for-purpose, longer-term/continual approach(es) to evaluation. Real-world research should reflect real-world evaluation and allow for the accrual of a narrative of value indicators that build a picture of the contribution of infrastructure to research outcomes. The linear approach is not fit for purpose, the research endeavour is a complex, twisted road, and the evaluation approach needs to embrace this complexity through the development of realist approaches and the rapidly evolving data ecosystem. This paper sets out methodological challenges and considers the need to develop bespoke methodological approaches to allow a richer assessment of impact, contribution, attribution, and evaluation of research infrastructure. This paper is the beginning of a conversation that invites the community to "take up the mantle" and tackle the complexity of real-world research translation and evaluation.

21 Code of Federal Regulations Part 11-Compliant Digital Signature Solution for Cancer Clinical Trials: A Single-Institution Feasibility Study

PURPOSE

Inefficiencies in the clinical trial infrastructure result in protracted trial completion timelines, physician-investigator turnover, and a shrinking skilled labor force and present obstacles to research participation. Taken together, these barriers hinder scientific progress. Technological solutions to improve clinical trial efficiency have emerged, yet adoption remains slow because of concerns with cost, regulatory compliance, and implementation.

METHODS

A prospective pilot study that compared regulatory-compliant digital and traditional wet ink paper signatures was conducted over a 6.5-month period in a hospital-based health system. Staff time and effort, error rate, costs, and time to completion were measured. Wilcoxon rank sum tests were used to compare staff time and time to completion. A value analysis was conducted. A survey was administered to measure user satisfaction.

RESULTS

There where 96 participants (47 digital, 49 paper), 132 studies included (31 digital, 101 paper), and 265 documents processed (156 digital, 109 paper). A moderate reduction in staff time required to prepare documents for signature was observed (P < .0001). Error rates were reported in 5.1% of digital and 2.8% of paper documents, but this difference was not significant. Discrepancies requiring revisions included incomplete mandatory fields, inaccurate information submitted, and technical issues. A value analysis demonstrated a 19% labor savings with the use of digital signatures. Survey response rate was 57.4% (n = 27). Most participants (85.2%) preferred digital signatures. The time to complete documents was faster with digital signatures compared with paper (P = .0241).

CONCLUSION

The use of digital signatures resulted in a decrease in document completion time and regulatory burden as represented by staff hours. Additional cost and time savings and information liquidity could be realized by integrating digital signatures and electronic document management systems.

Trust in Community-Engaged Research Partnerships: A Methodological Overview of Designing a Multisite Clinical and Translational Science Awards (CTSA) Initiative

Community-engaged research (CEnR) builds on the strengths of the Clinical and Translational Science Awards (CTSA) framework to address health in underserved and minority communities. There is a paucity of studies that identify the process from which trust develops in CEnR partnerships. This study responds to the need for empirical investigation of building and maintaining trust from a multistakeholder perspective. We conducted a multi-institutional pilot study using concept mapping with to better understand how trust, a critical outcome of CEnR partnerships, can act as "social capital." Concept mapping was used to collect data from the three stakeholder groups: community, health-care, and academic research partners across three CTSAs. Concept mapping is a mixed-methods approach that allows participants to brainstorm and identify factors that contribute to a concept and describe ways in which those factors relate to each other. This study offers important insights on developing an initial set of trust measures that can be used across CTSAs to understand differences and similarities in conceptualization of trust among key stakeholder groups, track changes in public trust in research, identify both positive and negative aspects of trust, identify characteristics that maintain trust, and inform the direction for future research.

Activating clinical trials: a process improvement approach

Background

The administrative process associated with clinical trial activation has been criticized as costly, complex, and time-consuming. Prior research has concentrated on identifying administrative barriers and proposing various solutions to reduce activation time, and consequently associated costs. Here, we expand on previous research by incorporating social network analysis and discrete-event simulation to support process improvement decision-making.

Methods

We searched for all operational data associated with the administrative process of activating industry-sponsored clinical trials at the Office of Clinical Research of the University of South Florida in Tampa, Florida. We limited the search to those trials initiated and activated between July 2011 and June 2012. We described the process using value stream mapping, studied the interactions of the various process participants using social network analysis, and modeled potential process modifications using discrete-event simulation.

Results

The administrative process comprised 5 sub-processes, 30 activities, 11 decision points, 5 loops, and 8 participants. The mean activation time was 76.6 days. Rate-limiting sub-processes were those of contract and budget development. Key participants during contract and budget development were the Office of Clinical Research, sponsors, and the principal investigator. Simulation results indicate that slight increments on the number of trials, arriving to the Office of Clinical Research, would increase activation time by 11 %. Also, incrementing the efficiency of contract and budget development would reduce the activation time by 28 %. Finally, better synchronization between contract and budget development would reduce time spent on batching documentation; however, no improvements would be attained in total activation time.

Conclusion

The presented process improvement analytic framework not only identifies administrative barriers, but also helps to devise and evaluate potential improvement scenarios. The strength of our framework lies in its system analysis approach that recognizes the stochastic duration of the activation process and the interdependence between process activities and entities.

Electronic supplementary material

The online version of this article (doi:10.1186/s13063-016-1227-2) contains supplementary material, which is available to authorized users.

Evaluation and the NIH clinical and translational science awards: a "top ten" list

Since 2006, a total of 61 Clinical and Translational Science Institutes (CTSAs) have been funded by the National Institutes of Health (NIH), with the aim of reducing translation time from a bench discovery to when it impacts patients. This special issue of Evaluation & the Health Professions focuses on evaluation within and across the large, complex system of the CTSA Program of NIH. Through insights gained by reading the articles in this special edition and the experience of the authors, a "top ten" list of lessons learned and insights gained is presented. The list outlines issues that face those who evaluate the influence of the CTSA Program, as they work to anticipate what will be needed for continuing success. Themes include (1) considering the needs of stakeholders, (2) the perspective of the evaluators, (3) the importance of service improvement, (4) the importance of teams and people, (5) costs and return on investments, (6) methodology considerations to evaluate the CTSA enterprise, (7) innovation in evaluation, (8) defining the transformation of research, (9) evaluating the long-term impact of the CTSAs on public health, and (10) contributing to science policy formulation and implementation. The establishment of the CTSA Program, with its mandated evaluation component, has not only influenced the infrastructure and nature of translational research but will continue to impact policy and management in science.