Pragmatic trials and implementation science: grounds for divorce?

Similarities and Differences Between Pragmatic Trials and Hybrid Effectiveness-Implementation Trials

Pragmatism in clinical trials is focused on increasing the generalizability of research findings for routine clinical care settings. Hybridism in clinical trials (i.e., assessing both clinical effectiveness and implementation success) is focused on speeding up the process by which evidence-based practices are developed and adopted into routine clinical care. Even though pragmatic trial methodologies and implementation science evolved from very different disciplines, Pragmatic Trials and Hybrid Effectiveness-Implementation Trials share many similar design features. In fact, these types of trials can easily be conflated, creating the potential for investigators to mislabel their trial type or mistakenly use the wrong trial type to answer their research question. Blurred boundaries between trial types can hamper the evaluation of grant applications, the scientific interpretation of findings, and policy-making. Acknowledging that most trials are not pure Pragmatic Trials nor pure Hybrid Effectiveness-Implementation Trials, there are key differences in these trial types and they answer very different research questions. The purpose of this paper is to clarify the similarities and differences of these trial types for funders, researchers, and policy-makers. In addition, recommendations are offered to help investigators choose, label, and operationalize the most appropriate trial type to answer their research question. These recommendations complement existing reporting guidelines for clinical effectiveness trials (TIDieR) and implementation trials (StaRI).

Strategies to implement evidence-informed decision making at the organizational level: a rapid systematic review

BACKGROUND

Achievement of evidence-informed decision making (EIDM) requires the integration of evidence into all practice decisions by identifying and synthesizing evidence, then developing and executing plans to implement and evaluate changes to practice. This rapid systematic review synthesizes evidence for strategies for the implementation of EIDM across organizations, mapping facilitators and barriers to the COM-B (capability, opportunity, motivation, behaviour) model for behaviour change. The review was conducted to support leadership at organizations delivering public health services (health promotion, communicable disease prevention) to drive change toward evidence-informed public health.

METHODS

A systematic search was conducted in multiple databases and by reviewing publications of key authors. Articles that describe interventions to drive EIDM within teams, departments, or organizations were eligible for inclusion. For each included article, quality was assessed, and details of the intervention, setting, outcomes, facilitators and barriers were extracted. A convergent integrated approach was undertaken to analyze both quantitative and qualitative findings.

RESULTS

Thirty-seven articles are included. Studies were conducted in primary care, public health, social services, and occupational health settings. Strategies to implement EIDM included the establishment of Knowledge Broker-type roles, building the EIDM capacity of staff, and research or academic partnerships. Facilitators and barriers align with the COM-B model for behaviour change. Facilitators for capability include the development of staff knowledge and skill, establishing specialized roles, and knowledge sharing across the organization, though staff turnover and subsequent knowledge loss was a barrier to capability. For opportunity, facilitators include the development of processes or mechanisms to support new practices, forums for learning and skill development, and protected time, and barriers include competing priorities. Facilitators identified for motivation include supportive organizational culture, expectations for new practices to occur, recognition and positive reinforcement, and strong leadership support. Barriers include negative attitudes toward new practices, and lack of understanding and support from management.

CONCLUSION

This review provides a comprehensive analysis of facilitators and barriers for the implementation of EIDM in organizations for public health, mapped to the COM-B model for behaviour change. The existing literature for strategies to support EIDM in public health illustrates several facilitators and barriers linked to realizing EIDM. Knowledge of these factors will help senior leadership develop and implement EIDM strategies tailored to their organization, leading to increased likelihood of implementation success.

REVIEW REGISTRATION

PROSPERO CRD42022318994.

From pilot to a multi-site trial: refining the Early Detection of Deterioration in Elderly Residents (EDDIE +) intervention

BACKGROUND

Early Detection of Deterioration in Elderly Residents (EDDIE +) is a multi-modal intervention focused on empowering nursing and personal care workers to identify and proactively manage deterioration of residents living in residential aged care (RAC) homes. Building on successful pilot trials conducted between 2014 and 2017, the intervention was refined for implementation in a stepped-wedge cluster randomised trial in 12 RAC homes from March 2021 to May 2022. We report the process used to transition from a small-scale pilot intervention to a multi-site intervention, detailing the intervention to enable future replication.

METHODS

The EDDIE + intervention used the integrated Promoting Action on Research Implementation in Health Services (i-PARIHS) framework to guide the intervention development and refinement process. We conducted an environmental scan; multi-level context assessments; convened an intervention working group (IWG) to develop the program logic, conducted a sustainability assessment and deconstructed the intervention components into fixed and adaptable elements; and subsequently refined the intervention for trial.

RESULTS

The original EDDIE pilot intervention included four components: nurse and personal care worker education; decision support tools; diagnostic equipment; and facilitation and clinical support. Deconstructing the intervention into core components and what could be flexibly tailored to context was essential for refining the intervention and informing future implementation across multiple sites. Intervention elements considered unsustainable were updated and refined to enable their scalability. Refinements included: an enhanced educational component with a greater focus on personal care workers and interactive learning; decision support tools that were based on updated evidence; equipment that aligned with recipient needs and available organisational support; and updated facilitation model with local and external facilitation.

CONCLUSION

By using the i-PARIHS framework in the scale-up process, the EDDIE + intervention was tailored to fit the needs of intended recipients and contexts, enabling flexibility for local adaptation. The process of transitioning from a pilot to larger scale implementation in practice is vastly underreported yet vital for better development and implementation of multi-component interventions across multiple sites. We provide an example using an implementation framework and show it can be advantageous to researchers and health practitioners from pilot stage to refinement, through to larger scale implementation.

TRIAL REGISTRATION

The trial was prospectively registered with the Australia New Zealand Clinical Trial Registry (ACTRN12620000507987, registered 23/04/2020).

Process evaluations of health-promotion interventions in sports settings: a systematic review

Sports settings have been identified as an ideal place to conduct complex multi-level health-promotion interventions, with the potential to engage a broad audience. Whilst the benefits of delivering health-promotion interventions in sports settings are well documented, such interventions' real-world implementation and success must be better understood. Process evaluations can be conducted to provide information related to an intervention's fidelity, replication, scaling, adoption, and the underlying mechanisms driving outcomes. This systematic review summarizes how process evaluations of health-promotion interventions are conducted in sports settings and highlight facilitators and barriers to health-promotion intervention delivery using narrative synthesis. Following the Preferred Reporting Items for Systematic Reviews and Meta-Analysis guidelines, searches included original peer-reviewed articles from inception-January 2023. We searched eight electronic databases: Academic Search Complete; MEDLINE, PsycARTICLES; PsycINFO; SPORTSDiscus with Full Text; MEDLINE; SCOPUS; Pub Med, and Pro Quest Central. Thirty-two studies were included. Findings suggest that most process evaluations of health-promotion interventions have acknowledged the inherent complexity of sports settings, and investigated factors that explain their intervention's success (e.g. trust building, engagement). However, poor use of process evaluation frameworks or guidelines resulted in wide variations of how process evaluations are conducted and reported, which made findings difficult to integrate and standardize with consistency. Accordingly, this review provides a guide on how future process evaluations can be conducted to improve health-promotion interventions' transparency, replicability and reliability in real-world settings.

Attrition one year after starting antiretroviral therapy before and after the programmatic implementation of HIV "Treat All" in Sub-Saharan Africa: a systematic review and meta-analysis

INTRODUCTION

Evidence on the real-world effects of "Treat All" on attrition has not been systematically reviewed. We aimed to review existing literature to compare attrition 12 months after antiretroviral therapy (ART) initiation, before and after "Treat All" was implemented in Sub-Saharan Africa and describe predictors of attrition.

METHODS

We searched Embase, Google Scholar, PubMed, and Web of Science in July 2020 and created alerts up to the end of June 2023. We also searched for preprints and conference abstracts. Two co-authors screened and selected the articles. Risk of bias was assessed using the modified Newcastle-Ottawa Scale. We extracted and tabulated data on study characteristics, attrition 12 months after ART initiation, and predictors of attrition. We calculated a pooled risk ratio for attrition using random-effects meta-analysis.

RESULTS

Eight articles and one conference abstract (nine studies) out of 8179 screened records were included in the meta-analysis. The random-effects adjusted pooled risk ratio (RR) comparing attrition before and after "Treat All" 12 months after ART initiation was not significant [RR = 1.07 (95% Confidence interval (CI): 0.91-1.24)], with 92% heterogeneity (I2). Being a pregnant or breastfeeding woman, starting ART with advanced HIV, and starting ART within the same week were reported as risk factors for attrition both before and after "Treat All".

CONCLUSIONS

We found no significant difference in attrition before and after "Treat All" one year after ART initiation. While "Treat All" is being implemented widely, differentiated approaches to enhance retention should be prioritised for those subgroups at risk of attrition.

PROSPERO NUMBER

CRD42020191582 .

Promoting Evidence-Based Practice: The Influence of Novel Structural Change to Accelerate Translational Rehabilitation

OBJECTIVE

To evaluate changes in clinicians' use of evidence-based practice (EBP), openness toward EBP, and their acceptance of organizational changes after a rehabilitation hospital transitioned to a new facility designed to accelerate clinician-researcher collaborations.

DESIGN

Three repeated surveys of clinicians before, 7-9 months, and 2.5 years after transition to the new facility.

SETTING

Inpatient rehabilitation hospital.

PARTICIPANTS

Physicians, nurses, therapists, and other health care professionals (n=410, 442, and 448 respondents at Times 1, 2, and 3, respectively).

INTERVENTIONS

Implementation of physical (architecture, design) and team-focused (champions, leaders, incentives) changes in a new model of care to promote clinician-researcher collaborations.

MAIN OUTCOME MEASURES

Adapted versions of the Evidence-Based Practice Questionnaire (EBPQ), the Evidence-Based Practice Attitudes Scale (EBPAS), and the Organizational Change Recipients' Beliefs Scale (OCRBS) were used. Open-ended survey questions were analyzed through exploratory content analysis.

RESULTS

Response rates at Times 1, 2, and 3 were 67% (n=410), 69% (n=422), and 71% (n=448), respectively. After accounting for familiarity with the model of care, there was greater reported use of EBP at Time 3 compared with Time 2 (adjusted meant2=3.51, standard error (SE)=0.05; adj. meant3=3.64, SE=0.05; P=.043). Attitudes toward EBPs were similar over time. Acceptance of the new model of care was lower at Time 2 compared with Time 1, but rebounded at Time 3 (adjusted meant1=3.44, SE=0.04; adj. meant2=3.19, SE=0.04; P<.0001; adj. meant3=3.51, SE=0.04; P<.0001). Analysis of open-ended responses suggested that clinicians' optimism for the model of care was greater over time, but continued quality improvement should focus on cultivating communication between clinicians and researchers.

CONCLUSIONS

Accelerating clinician-researcher collaborations in a rehabilitation setting requires sustained effort for successful implementation beyond novel physical changes. Organizations must be responsive to clinicians' changing concerns to adapt and sustain a collaborative translational medicine model and allow sufficient time, probably years, for such transitions to occur.

Exercise Programming Modelling a Standard of Care Approach Improves Physical Health and Patient-Reported Outcomes in Individuals Living with Breast Cancer: A Pilot Study

Controlled study designs widely report that exercise improves the health of individuals living with breast cancer. Still, many individuals living with breast cancer are not active enough to experience the benefits of exercise. The Activating Cancer Communities through an Exercise Strategy for Survivors study was developed to reach more individuals living with cancer. This report describes the effects of a 12-week individualized exercise program that models a standard-of-care approach on body composition, physical fitness, and patient-reported outcomes in individuals living with breast cancer. Individuals living with breast cancer were recruited for the study and completed an exercise program twice weekly overseen by a Clinical Exercise Physiologist. A total of 43 participants completed the exercise intervention, and 36 withdrew from the study. All participants had significantly improved aerobic fitness, waist circumference, hip circumference, lower body endurance, physical activity behaviour, health-related quality of life, emotional status, and fatigue levels after completing the program. Flexibility, balance, and sleep scores did not change. The results from the 12-week individualized exercise program largely align with the results from more controlled study designs. These results support future initiatives integrating exercise therapy into the standard of care for individuals living with breast cancer.

mHealth intervention delivered in general practice to increase physical activity and reduce sedentary behaviour of patients with prediabetes and type 2 diabetes (ENERGISED): rationale and study protocol for a pragmatic randomised controlled trial

BACKGROUND

The growing number of patients with type 2 diabetes and prediabetes is a major public health concern. Physical activity is a cornerstone of diabetes management and may prevent its onset in prediabetes patients. Despite this, many patients with (pre)diabetes remain physically inactive. Primary care physicians are well-situated to deliver interventions to increase their patients' physical activity levels. However, effective and sustainable physical activity interventions for (pre)diabetes patients that can be translated into routine primary care are lacking.

METHODS

We describe the rationale and protocol for a 12-month pragmatic, multicentre, randomised, controlled trial assessing the effectiveness of an mHealth intervention delivered in general practice to increase physical activity and reduce sedentary behaviour of patients with prediabetes and type 2 diabetes (ENERGISED). Twenty-one general practices will recruit 340 patients with (pre)diabetes during routine health check-ups. Patients allocated to the active control arm will receive a Fitbit activity tracker to self-monitor their daily steps and try to achieve the recommended step goal. Patients allocated to the intervention arm will additionally receive the mHealth intervention, including the delivery of several text messages per week, with some of them delivered just in time, based on data continuously collected by the Fitbit tracker. The trial consists of two phases, each lasting six months: the lead-in phase, when the mHealth intervention will be supported with human phone counselling, and the maintenance phase, when the intervention will be fully automated. The primary outcome, average ambulatory activity (steps/day) measured by a wrist-worn accelerometer, will be assessed at the end of the maintenance phase at 12 months.

DISCUSSION

The trial has several strengths, such as the choice of active control to isolate the net effect of the intervention beyond simple self-monitoring with an activity tracker, broad eligibility criteria allowing for the inclusion of patients without a smartphone, procedures to minimise selection bias, and involvement of a relatively large number of general practices. These design choices contribute to the trial's pragmatic character and ensure that the intervention, if effective, can be translated into routine primary care practice, allowing important public health benefits.

TRIAL REGISTRATION

ClinicalTrials.gov (NCT05351359, 28/04/2022).

Challenges in evaluating implementation and effectiveness in real-world settings: evaluation proposal for school-based health-promoting intervention

There are various research designs and approaches to investigate how health-promoting activities are implemented in complex, real-world systems, and to identify potential health effects that might occur following implementation. Although literature describes guidelines to perform and report about implementation research and effect evaluations, no specific guidelines exist on analysing and reporting about the combination of effectiveness data and implementation data collected as part of intervention evaluation in complex and diverse settings. This paper describes the evaluation of primary school-based health-promoting activities in complex systems. Furthermore, an approach for data categorization inspired by Rogers' Diffusion of Innovations theory is presented that can facilitate structuring the study's results and relating the degree of implementation to any impact on effectiveness outcomes that might be observed. Researchers interested in using this approach for data categorization have to ensure that the following three conditions are met: (i) data on an intervention's efficacy in a controlled setting with optimal implementation is available; (ii) key points that define an intervention's optimal implementation are available and (iii) an evaluation study is performed, collecting both effectiveness data and implementation data in a real-world context. This data categorization approach can be useful to generate more insight into an intervention's effectiveness under varying circumstances, and optimal support and advice can be provided to stakeholders to achieve maximum impact of population-based health-promoting interventions in complex, real-world systems. However, the proposed approach is a first suggestion and further testing and adaptation is necessary to increase its usefulness. Knowledge and experience sharing among researchers performing comparable research can increase the knowledge base regarding this subject.

Understanding explanatory and pragmatic trials: Examples from randomized controlled trials on vertebroplasty

OBJECTIVES

To better understand the explanatory-pragmatic distinction in the design and interpretation of randomized controlled trials (RCTs).

METHODS

We review the explanatory-pragmatic distinction in clinical trial design. We use the PRECIS-2 tool to evaluate the trial design of selected RCTs on percutaneous vertebroplasty for osteoporotic vertebral compression fractures. We discuss difficulties in the selection of criteria and in the construction of PRECIS diagrams. We also examine how inconsistency in the selection of various items of trial design can cause confusion in the interpretation of results.

RESULTS

The selection of criteria and the scoring of multiple PRECIS domains were subjective and thus debatable. The pragmascope patterns of various vertebroplasty trials were heterogeneous. Many trials had both pragmatic and explanatory components. Some placebo-controlled trial goals seem to have been explanatory, but their design actually included enough pragmatic items such that the meaning of negative trial results remains ambiguous.

CONCLUSION

The results of a trial cannot be interpreted without understanding the various design choices made along the explanatory-pragmatic spectrum.

Integrated rapid-cycle comparative effectiveness trials using flexible point of care randomisation in electronic health record systems

Whilst the Randomised Controlled Trial remains the gold standard for deriving robust causal estimates of treatment efficacy, too often a traditional design proves prohibitively expensive or cumbersome when it comes to assessing questions regarding the comparative effectiveness of routinely used treatments. As a result, patients experience variation in practice as clinicians lack the evidence needed to personalise treatments effectively. This variation may be classified as unwarranted, where existing evidence is ignored, or legitimate where in the absence of evidence, clinicians rely on experience, expert opinion, and inferred principles from basic science to make decisions. We argue that within the right ethical and technological framework, legitimate variation can be transformed into a mechanism for evidence generation and learning. Learning Health Systems which harness existing variation in practice, represent a novel approach for generating evidence from everyday clinical practice. The development of these systems has gained traction due to the increased availability of modern Electronic Health Record Systems. However, despite their promise, overcoming hurdles to successfully integrating clinical trials within Learning Health Systems has proven challenging. This article describes the origins of integrated clinical trials and explores two main barriers to their further implementation - how best to obtain informed consent from patients to participate in routine comparative effectiveness research, and how to automate and integrate randomisation into a clinical workflow. Having described these barriers, we present a potential solution in the form of a research pipeline using a novel form of flexible point-of-care randomisation to allow clinicians and patients to participate in studies where there is clinical equipoise.

The science-policy relationship in times of crisis: An urgent call for a pragmatist turn

In this conceptual paper, we argue that at times of crisis, what is sometimes called "evidence-based" or "science-driven" policymaking-establishing scientific truths and then implementing them-must be tempered by a more agile, deliberative and inclusive approach which acknowledges and embraces uncertainty. We offer pragmatism as one potential option, using examples from the UK to illustrate how such an approach might have changed particular crisis decisions and led to better outcomes. We propose that to better prepare for the next public health crisis, five pragmatism-informed shifts are needed in the science-policy relationship: from scientism to science-informed narrative rationality that emerges from practice; from knowledge-then-action to acting judiciously under uncertainty; from hierarchies of evidence to pluralist inquiry; from polarized camps to frame-reflective dialogue; and from an "inside-track" science-policy dialogue to greater participatory democracy. We suggest an agenda for a pragmatist-informed program of applied research on crisis public health policymaking.

Implementing "from here to there": A case study of conceptual and practical challenges in implementation science

There is a significant challenge in global health and development research that pivots on the difficulties of delivering (cost-)effective treatments or interventions that are scalable andtransferable across settings. That is, how does one deliver "true effects", proven treatments, into new settings? This is often addressed in pragmatic trials or implementation research in which one makes adjustments to the delivery of the treatment to ensure that it works here and there. In this critical analytical review, we argue that the approach mis-characterises the cause-effect relationship and fails to recognise the local, highly contextual nature of what it means to say an intervention "works". We use an ongoing randomised controlled trial (RCT)-an informal settlement redevelopment intervention in Indonesia and Fiji to reduce human exposure to pathogenic faecal contamination-as a vehicle for exploring the ideas and implications of identifying interventions that work in global health and development. We describe the highly contextualised features of the research and the challenges these would pose in attempts to generalise the results. In other words, we detail that which is frequently elided from most RCTs. As our critical lens, we us the work of American philosopher, Nancy Cartwright, who argued that research produces dappled regions of causal insights-lacunae against a backdrop of causal ignorance. Rather than learn about a relationship between a treatment and an outcome, we learn that in the right sort of context, a treatment reliably produces a particular outcome. Moving a treatment from here to there becomes, therefore, something of an engineering exercise to ensure the right factors (or "shields") are in place so the cause-effect is manifest. As a consequence, one cannot assume that comparative effectiveness or cost-effectiveness would be maintained.

The Effects of a Multitiered System of Language Support on Kindergarten Oral and Written Language: A Large-Scale Randomized Controlled Trial

PURPOSE

The purpose of this randomized controlled trial was to examine the effects of a multitiered system of language support (MTSLS) on kindergarteners' narrative retelling, personal stories, writing, and expository language.

METHOD

Participants were 686 kindergarten students from four school districts in the United States. Twenty-eight classrooms were randomly assigned to treatment (n = 337 students) or control (n = 349 students) conditions. The treatment group received 14 weeks of oral narrative language instruction using Story Champs, a multitiered language program. Classroom teachers delivered large group (Tier 1) instruction for 15-20 min a day for 4 weeks. After this short-duration whole-class instruction, speech-language pathologists began small group Story Champs (Tier 2) intervention with a random sample of students who did not make adequate progress from the large group instruction (n = 49). These students received Tier 2 intervention for 20 min twice a week in addition to continued Tier 1 instruction.

RESULTS

Results indicated that the students in the treatment group had significantly higher scores on all outcome measures compared to the students in the control group. Analyses of outcomes from the 49 students who received Tier 2 intervention compared to a matching sample of at-risk control students revealed that the treatment group had significantly higher scores on narrative retells, personal stories, and expository retells. When compared to matched average-performing and advanced-performing control peers, the students who received Tier 2 intervention had significantly higher narrative retell scores and no longer had significantly lower personal story, expository, or writing scores.

CONCLUSION

This effectiveness study demonstrated that MTSLS can lead to meaningful improvements in kindergarteners' oral and written language skills, even helping at-risk students catch up to high-achieving peers.

Are morbidity and mortality estimates from randomized controlled trials externally valid? A comparison of outcomes among infants enrolled into an RCT or a cohort study in Botswana

Background

The external validity of the randomized controlled trial (RCT) refers to the extent to which the results of the RCT apply to the relevant, non-trial population and is impacted by its eligibility criteria, its organization, and its delivery of the intervention. Here, we compared the outcomes of mortality and hospitalization between an RCT and a cohort study that concurrently enrolled HIV-exposed uninfected (HEU) newborns in Botswana.

Methods

The Mpepu Study (the RCT) was a clinical trial which determined that co-trimoxazole (CTX) provided no survival benefit for HEUs, allowing both arms of the RCT to be used. The Maikaelelo study (the cohort study) was a prospective observational study that enrolled HEU newborns with telephone follow-up and no in-person visits. Rates of death and hospitalization in the pooled population, were modeled using cox-proportional hazards models for time to death or time to first hospitalization, with study setting (RCT vs. cohort study) as an independent variable. The causal effect of study setting on morbidity and mortality was obtained through a treatment effects approach.

Results

In total, 4,010 infants were included; 1,306 were enrolled into the cohort study and 2,704 were enrolled into the RCT. No significant differences in mortality were observed between the two study settings (HR: 1.28, 95% CI: 0.76, 2.13), but RCT participants had a lower risk of hospitalization (HR: 0.72, 95% CI: 0.58, 0.89) that decreased with age. However, RCT participants had a higher risk of hospitalization within the first six months of life. The causal risk difference in hospitalizations attributable to the RCT setting was -0.03 (95% CI: -0.05, -0.01).

Conclusions

Children in an RCT with rigorous application of national standard of care guidelines experienced a significantly lower risk of hospitalization than children participating in a cohort study that did not alter clinical care. Future research is needed to further investigate outcome disparities when real-world results fail to mirror those achieved in a clinical trial.

Trial registration

The Mpepu Trial was funded by the U.S. National Institutes of Health (No. NCT01229761) and the Maikaelelo Study was funded primarily by the U.S. Centers for Disease Control and Prevention (32AI007433-21).

A review of trauma and orthopaedic randomised clinical trials published in high-impact general medical journals

Introduction

The recent past has seen a significant increase in the number of trauma and orthopaedic randomised clinical trials published in “the big five” general medical journals. The quality of this research has, however, not yet been established.

Methods

We therefore set out to critically appraise the quality of available literature over a 10-year period (April 2010–April 2020) through a systematic search of these 5 high-impact general medical journals (JAMA, NEJM, BMJ, Lancet and Annals). A standardised data extraction proforma was utilised to gather information regarding: trial design, sample size calculation, results, study quality and pragmatism. Quality assessment was performed using the Cochrane Risk of Bias 2 tool and the modified Delphi list. Study pragmatism was assessed using the PRECIS-2 tool.

Results

A total of 25 studies were eligible for inclusion. Over half of the included trials did not meet their sample size calculation for the primary outcome, with a similar proportion of these studies at risk of type II error for their non-significant results. There was a high degree of pragmatism according to PRECIS-2. Non-significant studies had greater pragmatism that those with statistically significant results (p < 0.001). Only 56% studies provided adequate justification for the minimum clinically important difference (MCID) in the population assessed. Overall, very few studies were deemed high quality/low risk of bias.

Conclusions

These findings highlight that there are some important methodological concerns present within the current evidence base of RCTs published in high-impact medical journals. Potential strategies that may improve future trial design are highlighted.

Level of evidence

Level 1.

Supplementary Information

The online version contains supplementary material available at 10.1007/s00590-021-03137-3.

Protocol refinement for a diabetes pragmatic trial using the PRECIS-2 framework

Background

This report describes how we refined a protocol for a pragmatic comparative effectiveness study of two models of an evidence-based diabetes shared medical appointment intervention and used the PRECIS-2 rating system to evaluate these adaptations.

Methods

We report primary data collected between June and August 2019, and protocol refinements completed between 2018 and 2020. Twenty-two members of the study team collaborated in protocol refinement and completed the PRECIS-2 ratings of study pragmatism. We discuss study design refinements made to achieve the desired level of pragmatism vs. experimental control for each of the nine PRECIS-2 dimensions. Study team members received training on PRECIS-2 scoring and were asked to rate the study protocol on the nine PRECIS-2 dimensions. Ratings were compared using descriptive statistics.

Results

In general, the PRECIS-2 ratings revealed high levels of pragmatism, but somewhat less pragmatic ratings on the categories of Delivery and Organization (costs and resources). This variation was purposeful, and we provide the rationale for and steps taken to obtain the targeted level of pragmatism on each PRECIS-2 dimension, as well as detail design changes made to a) make the design more pragmatic and b) address COVID-19 issues. There was general agreement among team members and across different types of stakeholders on PRECIS-2 ratings.

Conclusions

We discuss lessons learned from use of PRECIS-2 and experiences in refining the study to be maximally pragmatic on some dimensions and less so on other dimensions. This paper expands on prior research by describing actions to achieve higher levels of pragmatism and revise our protocol fit to the changed context. We make recommendations for future use of PRECIS-2 to help address changing context and other strategies for the planning of and transparent reporting on pragmatic research and comparative effectiveness research.

Trial registration

Clinicaltrials.gov Registration ID: NCT03590041.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12913-021-07084-x.

Multimodal intervention for preventing peripheral intravenous catheter failure in adults (PREBACP): a multicentre, cluster-randomised, controlled trial

BACKGROUND

2 billion peripheral intravenous catheters (PIVC) are inserted into inpatients worldwide each year. Almost one in two PIVCs fail before completion of intravenous therapy. We aimed to determine the efficacy and costs of a multimodal intervention to reduce PIVC failure among hospitalised patients.

METHODS

PREBACP was a cluster-randomised, controlled trial done at seven public hospitals in Spain. Clusters (hospital wards) had at least 70% permanent staff and data were collected from patients aged 18 years and older with one or more PIVCs at the start of intravenous therapy. Clusters were randomly assigned (1:1) to the multimodal intervention or control group using a centralised, web-based randomisation software, and stratified by type of setting. We concealed randomisation to allocation, without masking patients or professionals to the intervention. An intervention using a multimodal model and dissemination of protocols, education for health-care professionals and patients, and feedback on performance was implemented for 12 months in the intervention group. The control group received usual care. The primary outcome was all-cause PIVC failure at 12 months (phlebitis, extravasation, obstruction, or infections). Subsequently, through an amendment to the protocol approved on July 25, 2021, we included dislodgement as part of PIVC failure. Analysis was by modified intention to treat, which included all randomly assigned hospital wards for whom data on the primary endpoint were available. This trial is registered with the ISRCTN Registry, ISRCTN10438530.

FINDINGS

Between Jan 1, 2019, and March 1, 2020, we randomly assigned 22 eligible clusters to receive the multimodal intervention (n=11 clusters; 2196 patients, 2235 PIVCs, and 131 nurses) or usual practice in the control group (n=11 clusters; 2282 patients, 2330 PIVCs, and 138 nurses). At 12 months, the proportion of PIVC failures was lower in the intervention group than in the control group (37·10% [SD 1·32], HR 0·81 [95% CI 0·72 to 0·92] vs 46·49% [2·59], HR 1·23 [1·04 to 1·39]; mean difference -9·39% [95% CI -11·22 to -7·57]; p<0·0001). Per-protocol-prespecified analysis of the primary outcome excluding dislodgement also showed the intervention significantly reduced PIVC failure compared with the control group at 12 months (33·47% [SD 2·98], HR 0·85 [95% CI 0·75 to 0·96] vs 41·06% [4·62], HR 1·18 [1·04 to 1·33]; mean difference -7·59% [95% CI -11·05 to -4·13]; p<0·0001).

INTERPRETATION

A multimodal intervention reduced PIVC failure, thereby reducing potentially serious complications for hospitalised patients. The findings of PREBACP enabled a deeper understanding of decision making, knowledge mobilisation, and sense making in routine clinical practice.

FUNDING

The College of Nurses of the Balearic Islands.

TRANSLATION

For the Spanish translation of the abstract see Supplementary Materials section.

Barriers impacting the POINT pragmatic trial: the unavoidable overlap between research and intervention procedures in "real-world" research

BACKGROUND

This manuscript provides a research update to the ongoing pragmatic trial of Project POINT (Planned Outreach, Intervention, Naloxone, and Treatment), an emergency department-based peer recovery coaching intervention for linking patients with opioid use disorder to evidence-based treatment. The research team has encountered a number of challenges related to the "real-world" study setting since the trial began. Using an implementation science lens, we sought to identify and describe barriers impacting both the intervention and research protocols of the POINT study, which are often intertwined in pragmatic trials due to the focus on external validity.

METHOD

Qualitative data were collected from 3 peer recovery coaches, 2 peer recovery coach supervisors, and 3 members of the research team. Questions and deductive qualitative analysis were guided by the Consolidated Framework for Implementation Research (CFIR).

RESULTS

Nine unique barriers were noted, with 5 of these barriers impacting intervention and research protocol implementation simultaneously. These simultaneous barriers were timing of intervention delivery, ineffective communication with emergency department staff, lack of privacy in the emergency department, the fast-paced emergency department setting, and patient's limited resources. Together, these barriers represent the intervention characteristics, inner setting, and outer setting domains of the CFIR.

CONCLUSION

Results highlight the utility of employing an implementation science framework to assess implementation issues in pragmatic trials and how this approach might be used as a quality assurance mechanism given the considerable overlap that exists between research and intervention protocols in real-world trial settings. Previously undocumented changes to the trial design that have been made as a result of the identified barriers are discussed.

Advancing clinical leadership to improve the implementation of evidence-based practice in surgery: a longitudinal mixed-method study protocol

BACKGROUND

Clinical leadership is fundamental in facilitating service improvements in healthcare. Few studies have attempted to understand or model the different approaches to leadership which are used when promoting the uptake and implementation of evidence-based interventions. This research aims to uncover and explain how distributed clinical leadership can be developed and improved to enhance the use of evidence in practice. In doing so, this study examines implementation leadership in orthopaedic surgery to explain leadership as a collective endeavour which cannot be separated from the organisational context.

METHODS

A mixed-method study consisting of longitudinal and cross-sectional interviews and an embedded social network analysis will be performed in six NHS hospitals. A social network analysis will be undertaken in each hospital to uncover the organisational networks, the focal leadership actors and information flows in each organisation. This will be followed by a series of repeated semi-structured interviews, conducted over 4 years, with orthopaedic surgeons and their professional networks. These longitudinal interviews will be supplemented by cross-sectional interviews with the national established surgical leaders. All qualitative data will be analysed using a constructivist grounded theory approach and integrated with the quantitative data. The participant narratives will enrich the social network to uncover the leadership configurations which exist, and how different configurations of leadership are functioning in practice to influence implementation processes and outcomes.

DISCUSSION

The study findings will facilitate understanding about how and why different configurations of leadership develop and under what organisational conditions and circumstances they are able to flourish. The study will guide the development of leadership interventions that are grounded in the data and aimed at advancing leadership for service improvement in orthopaedics. The strength of the study lies in the combination of multi-component, multi-site, multi-agent methods to examine leadership processes in surgery. The findings may be limited by the practical challenges of longitudinal qualitative data collection, such as ensuring participant retention, which need to be balanced against the theoretical and empirical insights generated through this comprehensive exploration of leadership across and within a range of healthcare organisations.

Dissemination and implementation research in sports and exercise medicine and sports physical therapy: translating evidence to practice and policy

Knowledge from research evidence is wasted unless it is applied. While the scientific evidence base for many sports and exercise medicine and sports physical therapy interventions is robust, real-world implementation and evolution to scale remains an ongoing challenge. Dissemination and implementation research is important to generate evidence-informed, cost-effective and context-specific strategies for implementation partners and stakeholders to effectively apply and sustain the best research evidence in public health and clinical practice. However, this field of inquiry remains underexplored in sports and exercise medicine and sports physical therapy. Most intervention studies in sports and exercise medicine and sports physical therapy are terminated at the efficacy trial stage without considerations for best practices for translation to community and clinical settings. Lack of context-specific dissemination and implementation strategies to drive the translation of evidence-based interventions results in poor execution of, and attrition from, interventions, and this is associated with suboptimal outcomes and increased healthcare costs. Theory-driven quality research informing the successful dissemination and implementation of evidence-based interventions is needed to address lingering evidence-to-practice gaps. Dissemination and implementation research completes the final stage in the research-to-practice pipeline. It seeks to close evidence-to-practice gaps, thereby ensuring speedy application of research evidence to achieve desired public health outcomes while making more efficient use of limited resources. This review introduces sports and exercise medicine and sports physical therapy researchers and stakeholders to key concepts and principles in dissemination and implementation research.

The importance of decision intent within descriptions of pragmatic trials

OBJECTIVE

It is now more than 50 years since the concepts of explanatory and pragmatic attitudes toward trials were first discussed by Schwartz and Lellouch in their influential 1967 paper. Since then, there has been increasing focus on design aspects that may be consistent with more pragmatic attitudes within clinical trials, and a number of tools developed to assist investigators prospectively think about their trial design. Researchers have subsequently expressed interest in using these tools retrospectively to characterize trials as pragmatic or explanatory.

RESULTS

We suggest that recent attempts to retrospectively dichotomize trials solely on the basis of quantitative scoring of trial design features are flawed. Instead, we argue that there is a need to consider both the intent and design when assessing the degree of pragmatism within a trial.

CONCLUSION

The practical implication of our suggestion for trial reporting is that investigators should explicitly state the intent of the trial through a clear articulation of the decision that they hope will be informed by the trial results. This should be coupled with a completed PRagmatic-Explanatory Continuum Indicator Summary 2 assessment (or similar) with an explanation of study design choices to appropriately assess whether the study design is consistent with the study intent. We believe this will assist reviewers and knowledge users in making assessments of trials.