Online Resources for Dissemination and Implementation Science: Meeting Demand and Lessons Learned

Usability Testing of a Web Tool for Dissemination and Implementation Science Models

Dissemination and Implementation science is dedicated to increasing the speed of evidence-based research translated into practice as guided by one or multiple D&I theories, models, and frameworks. The Dissemination and Implementation Models in Health Research and Practice web tool guides users on how to plan, select, combine, adapt, use, and assess theories, models, and frameworks. This paper describes usability testing to update the web tool. Iterative user testing was conducted with implementation science research and clinical participants to facilitate updates and optimize the functionality of the tool. A multi-step protocol involved quantitative and qualitative data collection including a survey, interviews, and a usability testing session. Data from the pre-testing surveys were summarized as frequencies. Data from the usability testing sessions were analyzed using a hybrid adapted deductive rapid matrix qualitative analysis. Data from the interviews were analyzed by deductive a priori coding. Fifteen interviewees represented different research and clinical groups and levels of expertise utilizing D&I TMFs. Participants were purposively selected to represent a range of disciplines and D&I expertise, all invited via one-time email. The 847 total interview comments were reduced by similarity to 259 comments, and 142 were feasible changes fitting the priorities of the web tool. Changes to content, format, and functionality are described in this paper. The iterative usability testing elicited improvements to the web tool including adding more examples, definitions, visuals, and tutorials and simplifying the written content. The web tool remains flexible for additions concerning health equity, de-implementation, and other issues.

Supplementary Information

The online version contains supplementary material available at 10.1007/s43477-024-00125-7.

Implementation science for cancer control: One center's experience addressing context, adaptation, equity, and sustainment

Implementation science (IS) has great potential to enhance the frequency, speed, and quality of the translation of evidence-based programs, policies, products, and guidelines into practice. Progress has been made, but with some notable exceptions, this promise has not been achieved for cancer prevention and control. We discuss five interrelated but conceptually distinct, crosscutting issues important to accelerate IS for cancer prevention and control and how our Colorado Implementation Science Center in Cancer Control (COISC3) addressed these issues. These needs and opportunities include more fully addressing changing, multi-level context; guiding rapid, iterative adaptations; evaluating innovative approaches to engagement and health equity; greater attention to costs and economic issues; and sustainability. We summarize conceptual issues; evaluation needs and capacity building activities and then provide examples of how our IS center addressed these five needs for cancer prevention and control. We discuss changes made to address priorities of (i) guiding adaptations of implementation strategies to address changing context and (ii) working on issues identified and prioritized by our primary care partners rather than the research team. We conclude with discussion of lessons learned, limitations, and directions for future research and practice in IS to enhance cancer prevention and control as well as translational behavioral medicine more generally.

Integrating a focus on health equity in implementation science: Case examples from the national cancer institute's implementation science in cancer control centers (ISC3) network

Background

A Health Equity Task Force (HETF) of members from seven Centers funded by the National Cancer Institute's (NCI) Implementation Science in Cancer Control Centers (ISC3) network sought to identify case examples of how Centers were applying a focus on health equity in implementation science to inform future research and capacity-building efforts.

Methods

HETF members at each ISC3 collected information on how health equity was conceptualized, operationalized, and addressed in initial research and capacity-building efforts across the seven ISC3 Centers funded in 2019-2020. Each Center completed a questionnaire assessing five health equity domains central to implementation science (e.g., community engagement; implementation science theories, models, and frameworks (TMFs); and engaging underrepresented scholars). Data generated illustrative examples from these five domains.

Results

Centers reported a range of approaches focusing on health equity in implementation research and capacity-building efforts, including (1) engaging diverse community partners/settings in making decisions about research priorities and projects; (2) applying health equity within a single TMF applied across projects or various TMFs used in specific projects; (3) evaluating health equity in operationalizing and measuring health and implementation outcomes; (4) building capacity for health equity-focused implementation science among trainees, early career scholars, and partnering organizations; and (5) leveraging varying levels of institutional resources and efforts to engage, include, and support underrepresented scholars.

Conclusions

Examples of approaches to integrating health equity across the ISC3 network can inform other investigators and centers' efforts to build capacity and infrastructure to support growth and expansion of health equity-focused implementation science.

Dissemination and implementation science resources, training, and scientific activities provided through CTSA programs nationally: Opportunities to advance D&I research and training capacity

Introduction

Clinical and Translational Science Award (CTSA) Program hubs are well-positioned to advance dissemination and implementation (D&I) research and training capacity nationally, though little is known about what D&I research support and services CTSAs provide. To address this gap, the CTSA Dissemination, Implementation, and Knowledge Transfer Working Group conducted an environmental scan of CTSAs (2017-2018).

Methods

Of 67 CTSA institutions, we contacted 43 that previously reported delivering D&I research services. D&I experts from these institutions were emailed a survey assessing D&I resources, services, training, and scientific projects. Responses were categorized and double-coded by study authors using a content analysis approach.

Results

Thirty-five of the 43 D&I experts (81.4%) responded. Challenges to CTSAs in developing and supporting D&I science activities were related to inadequate D&I science workforce (45.7%) and lack of understanding of D&I science (25.7%). Services provided included consultation/mentoring programs (68%), pilot funding/grants (50%), and workshops/seminars/conferences (46%). Training and workforce development in D&I were frequently identified as future priorities. Recommendations included increase training to meet demand (68.6%), accessible D&I tools/resources (34.3%), greater visibility/awareness of D&I methods (34.3%), consultation services (22.9%), and expand D&I science workforce (22.9%).

Conclusions

CTSAs have tremendous potential to support the advancement and impact of D&I science across the translational continuum. Despite the growing presence of D&I science in CTSAs, continued commitment and prioritization are needed from CTSA and institutional leadership to raise awareness of D&I science and its value, meet training demands, and develop necessary infrastructure for conducting D&I science.

Bootstrapping implementation research training: A successful approach for academic health centers

Demand for building competencies in implementation research (IR) outstrips supply of training programs, calling for a paradigm shift. We used a bootstrap approach to leverage external resources and create IR capacity through a novel 2-day training for faculty scientists across the four Texas Clinical & Translational Science Awards (CTSAs). The Workshop combined internal and external expertise, targeted nationally established IR competencies, incorporated new National Institutes of Health/National Cancer Institute OpenAccess online resources, employed well-known adult education principles, and measured impact. CTSA leader buy-in was reflected in financial support. Evaluation showed increased self-reported IR competency; statewide initiatives expanded. The project demonstrated that, even with limited onsite expertise, it was possible to bootstrap resources and build IR capacity de novo in the CTSA community.

A stakeholder engagement method navigator webtool for clinical and translational science

Stakeholder engagement is increasingly expected by funders and valued by researchers in clinical and translational science, yet many researchers lack access to expert consultation or training in selecting appropriate stakeholder engagement methods. We describe the development of a novel Stakeholder Engagement Navigator webtool. We conducted an environmental scan and literature review, along with investigator interviews, surveys, and engagement expert facilitated group discussion. We formally reviewed and cataloged 29 distinct engagement methods. To develop the webtool, we used an iterative design process that followed Design Thinking phases: empathize, define, ideate, prototype, and test. As prioritized during user testing, the Stakeholder Engagement Navigator webtool both educates and guides investigators in selecting an engagement method based on key criteria. The V1.0 Navigator webtool filters methods first by purpose of engagement (noted by 62% of users as the highest priority criteria), then by budget, time per stakeholder interaction, and total interactions. The tool is available at DICEMethods.org. The Stakeholder Engagement Navigator webtool is a user-centered, interactive webtool suitable for use by researchers seeking guidance on appropriate stakeholder engagement methods for clinical and translational research projects.

Roles dissemination and implementation scientists can play in supporting research teams

The field of dissemination and implementation (D&I) science is rapidly growing, with many scientists seeking to apply D&I science to enhance and expand the impact of their work. As the D&I field grows and collaborations of implementation scientists with other fields flourish, a description for the roles for D&I scientists as they collaborate with researchers from other fields could be beneficial. This paper exemplifies how the D&I scientist/researcher collaborative process might work and important elements to consider in doing so, as well as provide an outline on how collaborations might progress for different project needs. This is discussed through example scenarios to consider an implementation scientists' engagement in a research project and describe potential roles for implementation scientists in supporting research teams. We then discuss characteristics to consider when incorporating a D&I expert into a team and considerations in navigating the scenarios.

Surviving and Thriving as Physicians in General Internal Medicine Fellowship in the Twenty-First Century

General internal medicine (GIM) fellowships play an important role in the development of physician scientists and clinical educators, as well as leaders in academic medicine. Nevertheless, the challenges of developing another novel aspect to one's career, along with balancing coursework, research productivity, clinical duties, and personal life during fellowship, can be overwhelming. Similarly, successfully securing a job at the end of fellowship can be a daunting process. In this article, we discuss the foundational tenets and themes of the GIM fellowship. These themes include (1) finding your purpose and passion, with a focus on selecting research coursework and developing an area of study; (2) the role and importance of mentorship, including the various kinds of mentorship that fellows require (traditional and peer mentorship, sponsors, and coaches), as well as how to be an effective mentee; (3) securing research funding; (4) landing a job; (5) and protecting time to meet personal goals. There is an increased need for a vibrant, diverse, and successful generation of general internal medicine researchers to advance our understanding of complex issues in clinical medicine and healthcare delivery and to inform health policy. It is our hope that this piece helps to support that mission.

Building capacity in dissemination and implementation science: a systematic review of the academic literature on teaching and training initiatives

BACKGROUND

The field of dissemination and implementation (D&I) science has grown significantly over recent years. Alongside this, an increased demand for training in D&I from researchers and implementers has been seen. Research describing and evaluating D&I training opportunities, referred to here as 'capacity building initiatives' (CBIs), can help provide an understanding of different methods of training as well as training successes and challenges. However, to gain a more detailed understanding of the evidence-base and how D&I CBIs are being reported in publications, a field-wide examination of the academic literature is required.

METHODS

Systematic review to identify the type and range of D&I CBIs discussed and/or appraised in the academic literature. EMBASE, Medline and PsycINFO were searched between January 2006 and November 2019. Articles were included if they reported on a D&I CBI that was developed by the authors (of each of the included articles) or the author's host institution. Two reviewers independently screened the articles and extracted data using a standardised form.

RESULTS

Thirty-one articles (from a total of 4181) were included. From these, 41 distinct D&I CBIs were identified which focussed on different contexts and professions, from 8 countries across the world. CBIs ranged from short courses to training institutes to being part of academic programmes. Nearly half were delivered face-face with the remainder delivered remotely or using a blended format. CBIs often stipulated specific eligibility criteria, strict application processes and/or were oversubscribed. Variabilities in the way in which the D&I CBIs were reported and/or evaluated were evident.

CONCLUSIONS

Increasing the number of training opportunities, as well as broadening their reach (to a wider range of learners), would help address the recognised deficit in D&I training. Standardisation in the reporting of D&I CBIs would enable the D&I community to better understand the findings across different contexts and scientific professions so that training gaps can be identified and overcome. More detailed examination of publications on D&I CBIs as well as the wider literature on capacity building would be of significant merit to the field.

Developing implementation research capacity: longitudinal evaluation of the King's College London Implementation Science Masterclass, 2014-2019

BACKGROUND

Despite an increasing number of training opportunities in implementation science becoming available, the demand for training amongst researchers and practitioners is unmet. To address this training shortfall, we developed the King's College London 'Implementation Science Masterclass' (ISM), an innovative 2-day programme (and currently the largest of its kind in Europe), developed and delivered by an international faculty of implementation experts.

METHODS

This paper describes the ISM and provides delegates' quantitative and qualitative evaluations (gathered through a survey at the end of the ISM) and faculty reflections over the period it has been running (2014-2019).

RESULTS

Across the 6-year evaluation, a total of 501 delegates have attended the ISM, with numbers increasing yearly from 40 (in 2014) to 147 (in 2019). Delegates represent a diversity of backgrounds and 29 countries from across the world. The overall response rate for the delegate survey was 64.5% (323/501). Annually, the ISM has been rated 'highly' in terms of delegates' overall impression (92%), clear and relevant learning objectives (90% and 94%, respectively), the course duration (85%), pace (86%) and academic level 87%), and the support provided on the day (92%). Seventy-one percent of delegates reported the ISM would have an impact on how they approached their future work. Qualitative feedback revealed key strengths include the opportunities to meet with an international and diverse pool of experts and individuals working in the field, the interactive nature of the workshops and training sessions, and the breadth of topics and contexts covered.

CONCLUSIONS

Yearly, the UK ISM has grown, both in size and in its international reach. Rated consistently favourably by delegates, the ISM helps to tackle current training demands from all those interested in learning and building their skills in implementation science. Evaluation of the ISM will continue to be an annual iterative process, reflective of changes in the evidence base and delegates changing needs as the field evolves.

Scale-up of the Accrual to Clinical Trials (ACT) network across the Clinical and Translational Science Award Consortium: a mixed-methods evaluation of the first 18 months

Introduction

The Clinical and Translational Science Award (CTSA) Program is a Consortium of nearly 60 academic medical research centers across the USA and a natural network for evaluating the spread and uptake of translational research innovation across the Consortium.

Methods

Dissemination of the Accrual to Clinical Trials (ACT) Network, a federated clinical informatics data network for population-based cohort discovery, began January 2018 across the Consortium. Diffusion of innovation theory guided dissemination design and evaluation. Mixed-methods assessed the spread and uptake across the Consortium through July 1, 2019 (n = 48 CTSAs). Methods included prospective time activity tracking (Kaplan-Meier curves), and survey and qualitative interviews.

Results

Within 18 months, nearly 80% of CTSAs had joined the data network and two-thirds of CTSAs achieving technical readiness had initiated launch to local clinical investigators. Over 10,000 ACT Network queries are projected for 2019; and by 2020, nearly all CTSAs will have joined the network. Median time-from-technical-readiness-to-local-launch was 154 days (interquartile range: 87-225 days]. Quality improvement processes reduced time-to-launch by 35.2% (64 days, p = 0.0036). Lessons learned include: (1) conceptualize dissemination as two-stage adoption demonstrating value for both CTSA hub service providers and clinical investigators; (2) include institutional trial into dissemination strategies so CTSA hubs can refine internal workflows and gather local user feedback endorsement; (3) embrace designing-for-dissemination during technology development; and (4) sustain adaptive dissemination and customer relationship management to keep CTSA hubs and users engaged.

Conclusions

Scale-up and spread of the ACT Network provides lessons learned for others disseminating innovation across the CTSA Consortium. The Network is primed for embedded implementation research.

National Working Group on the RE-AIM Planning and Evaluation Framework: Goals, Resources, and Future Directions

The National Working Group on RE-AIM Planning and Evaluation Framework (herein Workgroup) was established in 2004 to support the application of the framework and advance dissemination and implementation science (D&I). Workgroup members developed and disseminated products and resources (and continue to do so) to advocate for consistent application of RE-AIM and allow for cross study comparisons. The purpose of this paper is to summarize key Workgroup activities, products, and services (e.g., webinars, consultations, planning tools) and enhance bidirectional communication between the Workgroup and RE-AIM users. The ultimate goal of this work is to serve as a forum for dissemination to improve the balance between RE-AIM user demand (needs) and the currently limited RE-AIM Workgroup supply (consultation and resources) to demonstrate and expand the utility of RE-AIM as a D&I planning and evaluation framework. A summary of resources is provided as well as specific examples of how the Workgroup has been responsive to user needs.

Assessing the Adoption of Recommended Standards, Novel Approaches, and Best Practices for Animal Health Surveillance by Decision Makers in Europe

Animal health surveillance is an important tool for disease mitigation and helps to promote animal health and welfare, protect human health, support efficient animal production, and enable trade. This study aimed to assess adoption of recommended standards and best practice for surveillance (including risk-based approaches) in Europe. It included scoping interviews with surveillance experts in Denmark, the Netherlands, Norway, and Switzerland to gather information on knowledge acquisition, decisions and implementation of surveillance, and perceptions. This was followed by an online survey among animal health and food safety surveillance users in EU, EEA, and Schengen countries. A total of 166 responses were collected from 27 countries; 111 were eligible for analysis. A strong preference for legislation and established standards was observed, with peer-reviewed publications, conferences, symposia, and workshops to be major sources of information. The majority of respondents indicated a need for international evaluation for surveillance and implied that considerations of cost-effectiveness were essential when making a decision to adopt new surveillance standards. However, most of the respondents did not use a formal evaluation to inform the adoption of new standards or only conducted a descriptive assessment before their implementation or adaptation. Only a few respondents reported a quantitative economic evaluation despite economic efficiency being considered as a highly relevant criterion for surveillance implementation. Constraints mentioned in the adoption of new surveillance standards included insufficient time, financial and human resources, and lack of competency. Researchers aiming to achieve impact by their surveillance work are advised to consider ways of influencing binding standards and to disseminate their work pro-actively using varied channels of engagement tailored to relevant target audiences and their needs. Generally, a more formal linkage between surveillance information and disease mitigation decisions-for example, by using systematic evaluation-could help increase the economic value of surveillance efforts. Finally, a collaborative, international platform for exchange and learning on surveillance as well as co-design and dissemination of surveillance standards is recommended.

Health Equity and Implementation Science in Heart, Lung, Blood, and Sleep-Related Research: Emerging Themes From the 2018 Saunders-Watkins Leadership Workshop

Disparities in health outcomes for heart, lung, blood, and sleep-related health conditions are pervasive in the United States, with an unequal burden experienced among structurally disadvantaged populations. One reason for this disparity is that despite the existence of effective interventions that promote health equity, few have been translated and implemented consistently in the healthcare system. To achieve health equity, there is a dire need to implement and disseminate effective evidence-based interventions that account for the complex and multilayered social determinants of health among marginalized groups across healthcare settings. To that end, the National Heart, Lung, and Blood Institute's Center for Translation Research and Implementation Science invited early stage investigators to participate in the inaugural Saunders-Watkins Leadership Workshop in May of 2018 at the National Institutes of Health. The goals of the workshop were to: (1) present an overview of health equity research, including areas which require ongoing investigation; (2) review how the fields of health equity and implementation science are related; (3) demonstrate how implementation science could be utilized to advance health equity; and (4) foster early stage investigator career success in heart, lung, blood, and sleep-related research. Herein, we highlight key themes from the 2-day workshop and offer recommendations for the future direction of health equity and implementation science research in the context of heart, lung, blood, and sleep-related health conditions.