Pragmatic measures: what they are and why we need them

Development and evaluation of the Capability, Opportunity, and Motivation to deliver Physical Activity in School Scale (COM-PASS)

BACKGROUND

Teachers are recognized as 'key agents' for the delivery of physical activity programs and policies in schools. The aim of our study was to develop and evaluate a tool to assess teachers' capability, opportunity, and motivation to deliver school-based physical activity interventions.

METHODS

The development and evaluation of the Capability, Opportunity, and Motivation to deliver Physical Activity in School Scale (COM-PASS) involved three phases. In Phase 1, we invited academic experts to participate in a Delphi study to rate, provide recommendations, and achieve consensus on questionnaire items that were based on the Capability, Opportunity, and Motivation Behavior (COM-B) model. Each item was ranked on the degree to which it matched the content of the COM-B model, using a 5-point scale ranging from '1 = Poor match' to '5 = Excellent match'. In Phase 2, we interviewed primary and secondary school teachers using a 'think-aloud' approach to assess their understanding of the items. In Phase 3, teachers (n = 196) completed the COM-PASS to assess structural validity using confirmatory factor analysis (CFA).

RESULTS

Thirty-eight academic experts from 14 countries completed three rounds of the Delphi study. In the first round, items had an average rating score of 4.04, in the second round 4.51, and in the third (final) round 4.78. The final tool included 14 items, which related to the six constructs of the COM-B model: physical capability, psychological capability, physical opportunity, social opportunity, reflective motivation, and automatic motivation. In Phase 2, ten teachers shared their interpretation of COM-PASS via a 20-min interview, which resulted in minor changes. In Phase 3, CFA of the 3-factor model (i.e., capability, opportunity, and motivation) revealed an adequate fit to the data (χ2 = 122.6, p < .001, CFI = .945, TLI = .924, RMSEA = .066). The internal consistencies of the three subscale scores were acceptable (i.e., capability: α = .75, opportunity: α = .75, motivation: α = .81).

CONCLUSION

COM-PASS is a valid and reliable tool for assessing teachers' capability, opportunity, and motivation to deliver physical activity interventions in schools. Further studies examining additional psychometric properties of the COM-PASS are warranted.

A case study in science-to-service application of epilepsy self-management

AIMS

Epilepsy self-management (ESM), the overall approach of reducing seizures and optimizing whole-health, is a targeted approach to improve population health for people with epilepsy (PWE). "Self-management for people with epilepsy and a history of negative health events" (SMART) is an 8-session group-format, remotely delivered ESM. This report describes the evolution of SMART development, testing and scale-up, taking advantage of ESM team expertise, community relationships and infrastructure established by social service agencies that deliver support to PWE.

METHODS

This is a case-study dissemination and implementation (D&I) science-to-service model using the RE-AIM framework approach (Reach, Effectiveness, Adoption, Implementation, and Maintenance) focused on 5 dimensions of individual- and setting-level outcomes important to program adoption, impact and sustainability. Performance evaluations include participation representativeness, ESM attendance and acceptability as well as change in relevant health outcomes.

RESULTS

SMART D & I is implemented via a collaboration of 3 unique regional, epilepsy-focused nonprofit social service organizations and a university team that developed SMART. The ongoing collaboration is expanding SMART delivery to PWE across 13 U.S. states. Thus far, we have trained 17 Nurse and Peer Educators (NEs and PEs). PEs (N = 10) have a mean age 51.1 (SD 10.4) years and a mean age of epilepsy diagnosis of 29.4 (SD 19.3). Of 128 participants offered SMART, and who provided age data (N = 86) mean age was 37.7 years (SD 14.4). Of participants who provided data on gender and race (N = 89), 65 were women (73.9 %), 18 African-American (20.2 %). Mean age of epilepsy diagnosis was 19.4 years (SD 16.6) and 59 (52.2 %) of PWE reported having seizures in the last 30 days pre-SMART sessions. Among those with attendance data (N = 103), mean number of SMART groups attended was 5.7 (SD 2.3). Mean values for past 30-day seizure frequency, 9-item Patient Health Questionnaire (PHQ-9), Generalized Anxiety Disorder Inventory (GAD-7) and 10-item Quality of Life in Epilepsy Scale (QOLIE-10) for PWE that provided both pre and post SMART data were 7.6 (SD 15.8) vs 2.8 (SD 3.4) p = 0.3, 7.63 (SD 6.6) vs 6.3 (SD 5.7) p = 0.95, 6.6 (SD 5.7) vs 6.67(SD 5.3) p = 0.47 and 2.8 (SD 0.8) vs 2.7 (SD 1.0) p = 0.07 respectively.

CONCLUSIONS

Implementing ESM using a RE-AIM/Iterative RE-AIM framework links intervention developers and community partners. While PWE have substantial barriers to health, including frequent seizures, they are able engage in the SMART program. Although a major limitation to patient-level evaluation is challenges in collecting post-SMART follow-up data, preliminary findings suggest a trend for improved quality of life.

Integrated Measure of PRogram Element SuStainability in Childcare Settings (IMPRESS-C): development and psychometric evaluation of a measure of sustainability determinants in the early childhood education and care setting

BACKGROUND

There is a need for valid and reliable measures of determinants of sustainability of public health interventions in early childhood education and care (ECEC) settings. This study aimed to develop and evaluate the psychometric and pragmatic properties of such a measure - the Integrated Measure of PRogram Element SuStainability in Childcare Settings (IMPRESS-C).

METHODS

We undertook a two-phase process guided by the COnsensus-based Standards for the selection of health status Measurement INstruments checklist (COSMIN) and Psychometric and Pragmatic Evidence Rating Scale (PAPERS). Phase 1 involved measure development; i.e., determining items and scales through an iterative process and assessment of face and content validity. Phase 2 involved the evaluation of psychometric and pragmatic properties. The 29-item measure completed by service executives (directors and nominated supervisors) was embedded in a larger survey from a national sample of Australian ECEC services assessing their implementation of nutrition and physical activity programs. Structural validity, concurrent validity, known groups validity, internal consistency, floor and ceiling effects, norms, and pragmatic qualities of the measure were assessed according to the PAPERS criteria.

RESULTS

The final measure contained 26 items, with respondents reporting how strongly they agreed or disagreed on a five-point Likert scale. Phase 1 assessments confirmed the relevance, and face and content validity of the scale. In Phase 2, we obtained 482 completed surveys, of which 84% (n = 405) completed the entire measure across 405 ECEC settings (one executive per service). Three of the four fit indices for the confirmatory factor analysis met the pre-specified criteria (SRMR = 0.056, CFI = 0.993, RMSEA = 0.067) indicating 'good' structural validity. The IMPRESS-C illustrated: 'good' internal consistency, with Cronbach's alpha values from 0.53 to 0.92; 'emerging' concurrent validity; 'poor' known groups validity; 'good' norms; and 'good' overall pragmatic qualities (cost, readability, length, and assessor burden).

CONCLUSIONS

The IMPRESS-C possesses strong psychometric and pragmatic qualities for assessing service executive-level perceptions of determinants influencing sustainment of public health interventions within ECEC settings. To achieve a full range of perspectives in this setting, future work should be directed to also develop and test measures of sustainability determinants at the implementer level (e.g., among individual educators and staff).

Defining a screening tool for post-traumatic stress disorder in East Africa: a penalized regression approach

Background

Scalable PTSD screening strategies must be brief, accurate and capable of administration by a non-specialized workforce.

Methods

We used PTSD as determined by the structured clinical interview as our gold standard and considered predictors sets of (a) Posttraumatic Stress Checklist-5 (PCL-5), (b) Primary Care PTSD Screen for the DSM-5 (PC-PTSD) and, (c) PCL-5 and PC-PTSD questions to identify the optimal items for PTSD screening for public sector settings in Kenya. A logistic regression model using LASSO was fit by minimizing the average squared error in the validation data. Area under the receiver operating characteristic curve (AUROC) measured discrimination performance.

Results

Penalized regression analysis suggested a screening tool that sums the Likert scale values of two PCL-5 questions-intrusive thoughts of the stressful experience (#1) and insomnia (#21). This had an AUROC of 0.85 (using hold-out test data) for predicting PTSD as evaluated by the MINI, which outperformed the PC-PTSD. The AUROC was similar in subgroups defined by age, sex, and number of categories of trauma experienced (all AUROCs>0.83) except those with no trauma history- AUROC was 0.78.

Conclusion

In some East African settings, a 2-item PTSD screening tool may outperform longer screeners and is easily scaled by a non-specialist workforce.

Ready or Not: A Conceptual Model of Organizational Readiness for Embedded Pragmatic Dementia Research

The National Institute on Aging Alzheimer's Disease/Alzheimer's Disease and Related Dementias Research Implementation Milestones emphasize the need for implementation research that maximizes up-take and scale-up of evidence-based dementia care practices across settings, diverse populations, and disease trajectories. Organizational readiness for implementation is a salient consideration when planning and conducting embedded pragmatic trials, in which interventions are implemented by provider staff. The current article examines the conceptual and theoretical underpinnings of organizational readiness for implementation and the operationalization of this construct. We offer a preliminary conceptual model for explicating and measuring organizational readiness and describe the unique characteristics and demands of implementing evidence-based interventions targeting persons with dementia and/or their care partners. [Research in Gerontological Nursing, 17(3), 149-160.].

Implementation of an Adapted Fidelity Scale for Pennsylvania Coordinated Specialty Care Programs for First Episode Psychosis

Coordinated Specialty Care (CSC) is an evidence-based model for early intervention of first episode psychosis (FEP). Monitoring fidelity to CSC models is essential for proper evaluation of program outcomes and quality improvement. To address variability across CSC programs and fidelity assessment-associated burden, an adapted fidelity scale was developed and implemented statewide in Pennsylvania. This report describes the design and preliminary performance of the adapted scale. The Pennsylvania FEP Fidelity Scale (PA-FEP-FS) assesses adherence to the CSC model by focusing on essential model components of other established fidelity scales, in tandem with program evaluation data. Initial data from fourteen PA-FEP sites from 2018 to 2021 were examined as preliminary validation. Assessment-associated burdens and costs were also estimated. PA-FEP-FS captured essential components of CSC models and proved feasible for implementation across varying program structures, while minimizing burden and cost. Programs utilized annual feedback as CSC benchmarks, evidenced by increased scores over time. PA-FEP-FS provides a model for adapting CSC fidelity scales to meet state- or local-level requirements while reducing assessment burdens and costs that may be associated with existing scales.

What do you think it means? Using cognitive interviewing to improve measurement in implementation science: description and case example

Pragmatic measures are essential to evaluate the implementation of evidence-based interventions. Cognitive interviewing, a qualitative method that collects partner feedback throughout measure development, is particularly useful for developing pragmatic implementation measures. Measure developers can use cognitive interviewing to increase a measure's fit within a particular implementation context. However, cognitive interviewing is underused in implementation research, where most measures remain "homegrown" and used for single studies. We provide a rationale for using cognitive interviewing in implementation science studies and illustrate its use through a case example employing cognitive interviewing to inform development of a measurement-based care protocol for implementation in opioid treatment programs. Applications of cognitive interviewing, including developing a common language with partners and collecting multi-level feedback on assessment procedures, to improve measurement in implementation science are discussed.

A Case Study of the Development of a Valid and Pragmatic Implementation Science Measure: The Barriers and Facilitators in Implementation of Task-Sharing Mental Health Interventions (BeFITS-MH) Measure

Background

Few implementation science (IS) measures have been evaluated for validity, reliability and utility - the latter referring to whether a measure captures meaningful aspects of implementation contexts. In this case study, we describe the process of developing an IS measure that aims to assess Barriers and Facilitators in Implementation of Task-Sharing in Mental Health services (BeFITS-MH), and the procedures we implemented to enhance its utility.

Methods

We summarize conceptual and empirical work that informed the development of the BeFITS-MH measure, including a description of the Delphi process, detailed translation and local adaptation procedures, and concurrent pilot testing. As validity and reliability are key aspects of measure development, we also report on our process of assessing the measure's construct validity and utility for the implementation outcomes of acceptability, appropriateness, and feasibility.

Results

Continuous stakeholder involvement and concurrent pilot testing resulted in several adaptations of the BeFITS-MH measure's structure, scaling, and format to enhance contextual relevance and utility. Adaptations of broad terms such as "program," "provider type," and "type of service" were necessary due to the heterogeneous nature of interventions, type of task-sharing providers employed, and clients served across the three global sites. Item selection benefited from the iterative process, enabling identification of relevance of key aspects of identified barriers and facilitators, and what aspects were common across sites. Program implementers' conceptions of utility regarding the measure's acceptability, appropriateness, and feasibility were seen to cluster across several common categories.

Conclusions

This case study provides a rigorous, multi-step process for developing a pragmatic IS measure. The process and lessons learned will aid in the teaching, practice and research of IS measurement development. The importance of including experiences and knowledge from different types of stakeholders in different global settings was reinforced and resulted in a more globally useful measure while allowing for locally-relevant adaptation. To increase the relevance of the measure it is important to target actionable domains that predict markers of utility (e.g., successful uptake) per program implementers' preferences. With this case study, we provide a detailed roadmap for others seeking to develop and validate IS measures that maximize local utility and impact.

The core functions and forms paradigm throughout EPIS: designing and implementing an evidence-based practice with function fidelity

There are numerous frameworks for implementing evidence-based practices (EBPs) in novel settings to achieve "fidelity." However, identifying appropriate referents for fidelity poses a challenge. The Core Functions and Forms paradigm offers a model that can inform adaptation decisions throughout all phases of the Exploration, Preparation, Implementation, Sustainment (EPIS) framework. We applied the Core Functions-Forms paradigm throughout the Exploration and Preparation phases of EPIS in the design of two EBPs targeting family protective factors among Latinos in San Diego, as well as describe plans for its use in Implementation and Sustainment. We employed a distinct approach for each intervention element to contrast adaptation decisions that prioritize adherence to either form or function fidelity. We describe our application of the functions-forms paradigm within the EPIS framework, focusing on the Preparation phase. We also provide functions-forms matrices that map out the relationship between individual intervention components (forms) and the essential processes (functions) by which components are theorized to exert their impact. This case study of how the core functions-forms framework can be mapped onto EPIS can support a conceptual shift from prioritizing form fidelity to also focusing on function fidelity. This might allow interventionists to target appropriate fidelity referents when adapting an EBP, rather than defaulting to maintaining fidelity to forms as described in the protocol. We see great promise for using this framework for guiding actions throughout all EPIS phases and informing future applications of this paradigm to foster more robust fidelity to function.

The Strong African American Families Program: Disrupting the Negative Consequences of Racial Discrimination Through Culturally Tailored, Family-Based Prevention

Racism continues to be a major source of stress for African Americans and can impair psychological functioning. Adolescents experiencing discrimination may engage in self-soothing, but risky behaviors, which leave them at risk for negative life trajectories. Black pride has been identified as a key factor in explaining the heterogeneity in responses to discrimination. Racial socialization, strategies parents use to promote Black pride and protect youth from discrimination, is an important focus of family-based prevention programs serving African American families. This study tests the efficacy of a culturally tailored preventive intervention for rural African American families to disrupt the negative consequences of discrimination on adolescent psychological functioning. Four waves of data from the Strong African American Families (SAAF) efficacy trial (Murry & Brody in Journal of Marital & Family Therapy 30(3):271-283, 2004) with 667 African American families in rural Georgia were used for this study. Structural equation modeling was used to test study hypotheses. Adolescent experiences with discrimination at age 15 predicted concurrent psychological functioning and multiple risk behaviors at age 16, including sexual risk behavior, substance use problems, academic failure, and juvenile justice involvement. Mediation analyses demonstrated that psychological functioning was a significant mediator of these relations. The SAAF program was associated with increases in racial socialization, which in turn fostered gains in adolescent Black pride. Black pride was indirectly associated with reduced risk behavior through adolescent psychological functioning, but Black pride did not moderate the effect of discrimination on psychological functioning. This study confirms that family-based prevention can support African American adolescent mental health in the context of discrimination. However, more emphasis on reducing exposure to discrimination is needed.

A case study of pragmatic measure development of the Treatment Integrity for Elementary Settings self-report measure for teachers

Introduction

Due to usability, feasibility, and acceptability concerns, observational treatment fidelity measures are often challenging to deploy in schools. Teacher self-report fidelity measures with specific design features might address some of these barriers. This case study outlines a community-engaged, iterative process to adapt the observational Treatment Integrity for Elementary Settings (TIES-O) to a teacher self-report version designed to assess the use of practices to support children's social-emotional competencies in elementary classrooms.

Method

Cognitive walkthrough interviews were conducted with teachers to improve the usability of the teacher self-report measure, called the Treatment Integrity for Elementary Schools-Teacher Report (TIES-T). Qualitative content analysis was used to extract themes from the interviews and inform changes to the measure.

Results

Increasing clarity and interactive elements in the measure training were the dominant themes, but suggestions for the measure format and jargon were also suggested.

Conclusion

The suggested changes resulted in a brief measure, training, and feedback system designed to support the teacher's use of practices to support children's social-emotional competencies in elementary classrooms. Future research with the TIES-T will examine the score reliability and validity of the measure.

Psychometric validation of the Perceived Perioperative Competence Scale-Revised Short Form

AIM

To develop a parsimonious, shortened version of the Perceived Perioperative Competence Scale-Revised for perioperative nurses to complete as part of their specialty training while retaining good psychometric properties.

DESIGN

A longitudinal online survey was adopted.

METHODS

A national sample of perioperative nurses from Australia completed an online survey at two different time points 6 months apart between February and October 2021. Confirmatory factor analysis was conducted for item reduction and construct validity, while criterion, convergent validity and internal consistency were examined.

RESULTS

Usable data for psychometric assessment were obtained from 485 operating room nurses at time 1 and 164 nurses at time 2. The original 40-item revised scale was reduced to an 18-item measure, maintaining its six domains. Cronbach's alpha for the 18-item scale was .92 at time 1 and .90 at time 2. Scale validation demonstrated moderate to weak positive relationships in perceived competence scores relative to general self-efficacy, years of operating room experience, postgraduate education and recency of training.

CONCLUSIONS

Results suggest the 18-item Perceived Perioperative Competence Scale-Revised Short Form has initial robust psychometric properties and may be implemented in clinical settings as part of perioperative transition-to-practice, orientation programs and yearly professional development reviews.

IMPLICATIONS FOR THE PROFESSION

This short-form scale can help prepare perioperative nurses to demonstrate clinical competence in a climate of increasing professional demands using a valid measure of competence required in clinical practice.

IMPACT

Short and validated scales of perioperative competence are needed in clinical practice. Assessment of practising operating room nurses' perceived competence is essential in quality care provision, workforce planning and human resource management. This study provides an 18-item measure of the previously validated 40-item Perceived Perioperative Competence Scale-Revised. This scale can help provide an option for future testing of perioperative nurses' competence in clinical or research settings.

PATIENT OR PUBLIC CONTRIBUTION

Perioperative nurses were involved in the design of the study, particularly in the assessment of validation of the tools used in the study.

Evaluating the pragmatic characteristics of advance care planning outcome measures in dementia clinical trials: A scoping review

BACKGROUND

Advance care planning (ACP) is a process that involves discussing a person's goals, values, and preferences; it is particularly important for persons living with dementia (PLWD) given that dementia is incurable and progressive. To ensure results that will impact real-world practices, ACP outcome measures must be psychometrically strong, meaningful to key partners, and pragmatic to collect. Therefore, we conducted a scoping review of outcome measures utilized in ACP randomized controlled clinical trials (RCTs) enrolling PLWD or their care partners and evaluated their pragmatic characteristics.

METHODS

We searched MEDLINE/PubMed, EMBASE, CINAHL, PsycINFO, and Web of Science for peer-reviewed ACP RCTs enrolling PLWD or their care partners from 2011 to 2021. We abstracted characteristics of primary and secondary outcome measures, including pragmatic characteristics using an adapted Psychometric and Pragmatic Evidence Rating Scale and ACP outcome domains using the standardized ACP Outcome Framework (i.e., process, action, healthcare, or quality of care).

RESULTS

We included 21 ACP RCTs. Trials included 103 outcome measures (39 primary and 64 secondary), of which 11% measured process, 14% measured action, 49% measured healthcare, and 26% measured quality of care. Twenty-four (23%) outcome measures were highly pragmatic, the majority of which (67%) reflected healthcare outcome measures. Sixty-one (59%) outcomes were assessed as highly relevant to PLWD or their care partners. Only 20% (n = 21) of outcome measures were embedded into clinical practice. Most (62%) RCTs were conducted in nursing homes, and 33% were focused PLWD with advanced stage disease.

CONCLUSIONS

In RCTs testing ACP interventions to support PLWD, only 23% of outcome measures were highly pragmatic, and most of these measured healthcare utilizations. Outcome assessments were rarely integrated into the EHR during routine clinical care. New outcome measures that address the lived experience of PLWD and their care partners plus have high pragmatic characteristics are needed for embedded pragmatic clinical trials.

CritCom: assessment of quality of interdisciplinary communication around deterioration in pediatric oncologic patients

Background

High-quality clinical care requires excellent interdisciplinary communication, especially during emergencies, and no tools exist to evaluate communication in critical care. We describe the development of a pragmatic tool focusing on interdisciplinary communication during patient deterioration (CritCom).

Methods

The preliminary CritCom tool was developed after a literature review and consultation with a multidisciplinary panel of global experts in communication, pediatric oncology, and critical care to review the domains and establish content validity iteratively. Face and linguistic validity were established through cognitive interviews, translation, and linguistic synthesis. We conducted a pilot study among an international group of clinicians to establish reliability and usability.

Results

After reviewing 105 potential survey items, we identified 52 items across seven domains. These were refined through cognitive interviews with 36 clinicians from 15 countries. CritCom was piloted with 433 clinicians (58% nurses, 36% physicians, and 6% other) from 42 hospitals in 22 countries. Psychometric testing guided the refinement of the items for the final tool. CritCom comprised six domains with five items each (30 total). The final tool has excellent reliability (Cronbach's alpha 0.81-0.86), usability (93% agree or strongly agree that the tool is easy to use), and similar performance between English and Spanish tools. Confirmatory factor analysis was used to establish the final 6-domain structure.

Conclusions

CritCom is a reliable and pragmatic bilingual tool to assess the quality of interdisciplinary communication around patient deterioration for children in diverse resource levels globally. Critcom results can be used to design and evaluate interventions to improve team communication.

Language and early literacy professional development: A complex intervention for early childhood educators and speech-language pathologists

Purpose: Improving language and literacy skills in preschoolers can lead to better life outcomes. One way speech-language pathologists (SLPs) can improve these skills in preschoolers is by supporting educators through professional development (PD). However, PD in early childhood education and care (ECEC) settings is a complex intervention. To improve preschoolers' language and literacy skills using PD, SLPs must first work with educators to change or increase educators' language and literacy-promoting behaviours. This paper aimed to describe educator behaviours and preschooler skills following a real-world language and literacy PD intervention facilitated by two community SLPs.Method: Two pragmatic studies were conducted across four ECEC centres: (1) an observation study of 13 educators' self-reported language and literacy promoting behaviours, and (2) a non-randomised controlled trial investigating the language and literacy skills of 82 preschoolers as reported by their educators and parents/carers.Result: After the intervention, educators rated themselves as performing language and literacy-promoting behaviours more frequently. Educators also rated the early reading skills of preschoolers more highly after the PD intervention, but not preschoolers' oral language or early writing skills. Parents/carers did not report any significant improvements in preschoolers' skills.Conclusion: PD as an SLP intervention, whilst promising, showed mixed outcomes. Educator outcomes improved; however, preschooler outcomes were varied.

Functional Assessment of Handwriting Among Children: A Systematic Review of the Psychometric Properties

IMPORTANCE

Handwriting legibility and speed assessments have a critical role in identifying and evaluating handwriting problems, which are common among children.

OBJECTIVE

The objective was to evaluate the psychometric properties and clinical utility of handwriting assessments for children ages 3 to 16 yr.

DATA SOURCES

A systematic review was conducted in CINAHL, PubMed (MEDLINE), Scopus, and education databases, with no time limits. The search strategy included a combination of the following keywords: handwriting, write, children, assessment, and validity. The exclusion criteria were assessment tools that were electronic, that focused on cognitive components of handwriting, or that only evaluated alphabets other than Latin.

STUDY SELECTION AND DATA COLLECTION

The systematic review was carried out on the basis of the COnsensus-based Standards for the selection of health Measurement INstruments (COSMIN) checklist and Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) recommendations. The protocol was registered in the Prospective Register of Systematic Reviews (PROSPERO).

FINDINGS

The 14 included instruments had a total sample of 4,987 children. Internal consistency ranged from moderate (.73; Writing Readiness Inventory Tool in Context) to high (.98; Letter Writing). The interexaminer reliability values of the 11 direct assessment instruments ranged from .77 (Systematic Screening for Handwriting Difficulties) to 1.00 (Handwriting Speed Test).

CONCLUSIONS AND RELEVANCE

In this systematic review, existing tools were evaluated by clinical utility and the quality of psychometric properties. Direct assessments showed good psychometric properties. Indirect and self-assessment tools demonstrated poor psychometric properties. Further research on screening tools and tools in other languages is needed. What This Article Adds: Specific learning disorders (e.g., dysgraphia) negatively affect academic learning and, when prolonged in time, self-concept. However, handwriting legibility and speed assessments could be used to identify and evaluate these learning disorders if an early referral to occupational therapy is carried out.

Thyroid cancer patient reported outcome measures in clinical practice: analysing acceptability and optimizing recruitment

BACKGROUND

Patient reported outcomes measures (PROMs) can provide valuable metrics in clinical trials and cancer registries. To ensure relevance, patient participation must be optimized and PROMs be highly acceptable to patients. There are few data reporting methods to maximize recruitment and a lack of consensus regarding appropriate PROMs for thyroid cancer survivors.

METHODS

All patients with a new diagnosis of thyroid (excluding micropapillary and anaplastic) cancer within a single Australian health district between January 2020 and December 2021 were invited to complete PROMs electronically, and self-report ease of use and comprehensiveness of each tool. Participants completed Short Form-12 (SF-12), European Organization of Research and Treatment of Cancer (EORTC-QLQ-C30), City of Hope Quality of Life-Thyroid Version (COH-TV) and Thyroid Cancer Quality of Life Survey (ThyCaQoL). Semi-structured qualitative telephone interviews explored patient priorities. An enhanced, multimodal recruitment strategy was instituted after 12 months due to low response rates.

RESULTS

Survey completion improved under enhanced recruitment (37/62, 60% versus 19/64, 30%, P = 0.0007) with no differences in demographic or clinical characteristics. Few (4%-7%) respondents rated surveys as difficult to complete. No single PROM comprehensively captured health-related quality of life, with disease-specific tools performing marginally better (54% ThyCaQoL and 52% CoH-TV) compared to generic tools (38% SF-12 and 42% EOROTC-QLQ-C30). Qualitative data suggested that concurrent diagnoses, and survey invitation prior to surgery, made surveys more difficult to complete.

CONCLUSION

A comprehensive and representative assessment of PROMs in thyroid cancer survivors requires the use of multiple survey tools and specialized staff to maximize recruitment.

Clinical tools to assess functional capacity before elective non-cardiac surgery: a scoping review protocol

OBJECTIVE

The objective of this scoping review is to map the evidence on clinical tools to assess functional capacity prior to elective non-cardiac surgery.

INTRODUCTION

Functional capacity is a strong prognostic indicator before surgery, which can be used to identify patients at elevated risk of postoperative complications, yet, there is no consensus on which clinical tools should be used to assess functional capacity in patients prior to non-cardiac surgery.

INCLUSION CRITERIA

This review will consider any randomized or non-randomized studies that evaluate the performance of a functional capacity assessment tool in adults (≥18 years) prior to non-cardiac surgery. For studies to be included, the tool must be used clinically for risk stratification. We will exclude studies on lung and liver transplant surgery, as well as ambulatory procedures performed under local anesthesia.

METHODS

The review will be conducted in line with the JBI methodology for scoping reviews. A peer-reviewed search strategy will be used to query relevant databases (ie, MEDLINE, Embase, EBM Reviews). Additional sources of evidence will include databases of non-peer-reviewed literature and the reference lists of included studies. Two independent reviewers will identify eligible studies in 2 stages: stage 1, based on titles and abstracts; and stage 2, based on full texts. Information on study details, measurement properties, pragmatic qualities, and/or clinical utility metrics will be charted in duplicate onto standardized data collection forms. The results will be presented using descriptive summaries, frequency tables, and visual plots that highlight the extent of evidence and remaining gaps in the validation process of each tool.

REVIEW REGISTRATION

Open Science Framework https://osf.io/6nfht.

A comparison of two group cognitive behavioral therapy protocols for anxiety in urban schools: appropriateness, child outcomes, and cost-effectiveness

Background

Cognitive behavioral therapy (CBT) for pediatric anxiety is efficacious for reducing anxiety symptoms and improving functioning, but many children are unable to access CBT for anxiety in community settings. Schools are an important setting in which children access mental health care, including therapy for anxiety. In this setting, therapy is usually delivered by Masters-level therapists.

Objectives

Friends for Life (FRIENDS), a 12-session, manualized, group CBT program for anxiety has demonstrated effectiveness when implemented in schools. However, prior research has also found challenges regarding feasibility and cultural fit when delivering FRIENDS in the urban school context. To address these challenges, we adapted FRIENDS for implementation in the school setting so that it might be more feasible and culturally appropriate for low-income, urban schools in the United States, while maintaining the core components of treatment. The current study uses a mixed-method approach to compare the effectiveness, cost-effectiveness, and perceived appropriateness of FRIENDS and CATS when delivered by Masters-level therapists with train-the-trainer support.

Materials and methods

First, we compared change scores for student outcomes (i.e., child-report MASC-2 total score, parent-report MASC-2 total score, teacher-report Engagement and Disaffection subscale scores) from pre- to post- treatment between students receiving FRIENDS and students receiving CATS to assess whether the two conditions resulted in equivalent outcomes. Second, we compared the cost and cost-effectiveness between the groups. Finally, we used an applied thematic analysis to compare appropriateness of the interventions as perceived by therapists and supervisors.

Results

The mean change score for the child-reported MASC-2 was 1.9 (SE = 1.72) points in the FRIENDS condition and 2.9 (SE = 1.73) points in the CATS condition; results indicated that the conditions were similar in their treatment effects, and symptom reductions were small in both groups. The modified protocol, CATS, was shown to cost significantly less to implement compared to FRIENDS and showed greater cost-effectiveness. Finally, compared to therapists and supervisors in the CATS condition, therapists and supervisors in the FRIENDS condition more strongly described aspects of the intervention that were not appropriate for their context and in need of more extensive adaptations.

Conclusion

Relatively brief, group CBT for anxiety, with adaptations to improve cultural fit, is a promising approach to treat youth anxiety symptom when delivered by school-based therapists with train-the-trainer implementation support.

The simplification of the insomnia severity index and epworth sleepiness scale using machine learning models

Insomnia and excessive daytime sleepiness (EDS) are the most common complaints in sleep clinics, and the cost of healthcare services associated with them have also increased significantly. Though the brief questionnaires such as the Insomnia Severity Index (ISI) and Epworth Sleepiness Scale (ESS) can be useful to assess insomnia and EDS, there are some limitations to apply for large numbers of patients. As the researches using the Internet of Things technology become more common, the need for the simplification of sleep questionnaires has been also growing. We aimed to simplify ISI and ESS using machine learning algorithms and deep neural networks with attention models. The medical records of 1,241 patients who examined polysomnography for insomnia or EDS were analyzed. All patients are classified into five groups according to the severity of insomnia and EDS. To develop the model, six machine learning algorithms were firstly applied. After going through normalization, the process with the CNN+ Attention model was applied. We classified a group with an accuracy of 93% even with only the results of 6 items (ISI1a, ISI1b, ISI3, ISI5, ESS4, ESS7). We simplified the sleep questionnaires with maintaining high accuracy by using machine learning models.

"We bleed for our community:" A qualitative exploration of the implementation of a pragmatic weight gain prevention trial from the perspectives of community health center professionals

BACKGROUND

Clinical trial implementation continues to shift toward pragmatic design, with the goal of increasing future adoption in clinical practice. Yet, few pragmatic trials within clinical settings have qualitatively assessed stakeholder input, especially from those most impacted by research implementation and outcomes, i.e., providers and staff. Within this context, we conducted a qualitative study of the implementation of a pragmatic digital health obesity trial with employees at a Federally qualified health center (FQHC) network in central North Carolina.

METHODS

Participant recruitment was conducted through purposive sampling of FQHC employees from a variety of backgrounds. Two researchers conducted semi-structured qualitative interviews and collected demographic data. Interviews were digitally recorded, professionally transcribed and double-coded by two independent researchers using NVivo 12. Coding discrepancies were reviewed by a third researcher until intercoder consensus was reached. Responses were compared within and across participants to elucidate emergent themes.

RESULTS

Eighteen qualitative interviews were conducted, of whom 39% provided direct medical care to patients and 44% worked at the FQHC for at least seven years. Results illuminated the challenges and successes of a pragmatically designed obesity treatment intervention within the community that serves medically vulnerable patients. Although limited time and staffing shortages may have challenged recruitment processes, respondents described early buy-in from leadership; an alignment of organizational and research goals; and consideration of patient needs as facilitators to implementation. Respondents also described the need for personnel power to sustain novel research interventions and considerations of health center resource constraints.

CONCLUSIONS

Results from this study contribute to the limited literature on pragmatic trials utilizing qualitative methods, particularly in community-based obesity treatment. To continue to merge the gaps between research implementation and clinical care, qualitative assessments that solicit stakeholder input are needed within pragmatic trial design. For maximum impact, researchers may wish to solicit input from a variety of professionals at trial onset and ensure that shared common goals and open collaboration between all partners is maintained throughout the trial.

TRIAL REGISTRATION

This trial was registered with ClinicalTrials.gov (NCT03003403) on December 28, 2016.

A Scoping Review on Quality Physical Education Programmes and Their Outcomes on Primary-Level Pupils

A scoping review was carried out on the literature relating to the evaluation of the implementation of quality physical education (QPE) programmes and related outcomes on final-stage primary-level pupils' attitudes towards physical education (ATPE), physical activity behaviour (PAB), mental wellbeing (MWB) and academic achievement (AA). The scoping review included studies published between 2000 and 2020 in the PubMed, Elsevier, SCOPUS and CINAHL databases and was completed in accordance with the PRISMA extension for scoping reviews' guidelines. Based on the inclusion criteria, 15 out of 2869 studies were included in the review. A thematic analysis was used to inductively and deductively analyse the studies for common themes of features of QPE programmes in primary schools, arising from nine different countries, considering the four outcome dimensions (ATPE, PAB, MWB and AA). The common themes identified as features of QPE across all four dimensions were as follows: (1) government leadership; (2) PE curriculum; (3) school principal and leaders; (4) organisational management from leadership in school; (5) teachers; (6) parental involvement; and (7) community partnerships. Based on these findings, recommendations were made for an evaluation framework on QPE in primary education.

Dissemination and implementation research for oral and craniofacial health: Background, a review of literature and future directions

Oral conditions are highly prevalent globally and have profound consequence on individuals and communities. Clinical (e.g. dental treatments, behavioural counselling) and non-clinical (e.g. community-based programming, water fluoridation, oral health policy) evidence-based interventions have been identified, recommended and applied at the clinic, community and policy levels. Still, the burden of oral conditions persists, with inequitable distribution across populations. A major driver of this lack of progress is poor translation of research findings, which results in an evidence-to-practice gap. Dissemination and implementation science (DIS) has emerged to address this gap. A relatively new field, application of DIS represents an important avenue for achieving good dental, oral and craniofacial health for all. The goal of this introductory article is to provide a brief background on DIS relevant to researchers in dentistry and oral health. The problem of knowledge translation, basic concepts and terminology in DIS, and approaches to doing dissemination and implementation research-including implementation strategies, key outcomes, and implementation theories, models and frameworks-are discussed. Additionally, the article reviews literature applying DIS to dentistry and oral health. Results of published studies and their implications for the field are presented. Drawing on the literature review and contemporary thinking in DIS, current gaps, opportunities and future directions are discussed. Resources for understanding and applying DIS are provided throughout. This article serves as a primer on DIS for dental and oral health researchers of all types working across a range of contexts; it also serves as a call to action for increased application of DIS to address the burden of oral conditions globally.

A pragmatic context assessment tool (pCAT): using a Think Aloud method to develop an assessment of contextual barriers to change

BACKGROUND

The Consolidated Framework for Implementation Research (CFIR) is a determinant framework that can be used to guide context assessment prior to implementing change. Though a few quantitative measurement instruments have been developed based on the CFIR, most assessments using the CFIR have relied on qualitative methods. One challenge to measurement is to translate conceptual constructs which are often described using highly abstract, technical language into lay language that is clear, concise, and meaningful. The purpose of this paper is to document methods to develop a freely available pragmatic context assessment tool (pCAT). The pCAT is based on the CFIR and designed for frontline quality improvement teams as an abbreviated assessment of local facilitators and barriers in a clinical setting.

METHODS

Twenty-seven interviews using the Think Aloud method (asking participants to verbalize thoughts as they respond to assessment questions) were conducted with frontline employees to improve a pilot version of the pCAT. Interviews were recorded and transcribed verbatim; the CFIR guided coding and analyses.

RESULTS

Participants identified several areas where language in the pCAT needed to be modified, clarified, or allow more nuance to increase usefulness for frontline employees. Participants found it easier to respond to questions when they had a recent, specific project in mind. Potential barriers and facilitators tend to be unique to each specific improvement. Participants also identified missing concepts or that were conflated, leading to refinements that made the pCAT more understandable, accurate, and useful.

CONCLUSIONS

The pCAT is designed to be practical, using everyday language familiar to frontline employees. The pCAT is short (14 items), freely available, does not require research expertise or experience. It is designed to draw on the knowledge of individuals most familiar with their own clinical context. The pCAT has been available online for approximately two years and has generated a relatively high level of interest indicating potential usefulness of the tool.

Tools for assessing health research partnership outcomes and impacts: a systematic review

OBJECTIVE

To identify and assess the globally available valid, reliable and acceptable tools for assessing health research partnership outcomes and impacts.

METHODS

We searched Ovid MEDLINE, Embase, CINAHL Plus and PsycINFO from origin to 2 June 2021, without limits, using an a priori strategy and registered protocol. We screened citations independently and in duplicate, resolving discrepancies by consensus and retaining studies involving health research partnerships, the development, use and/or assessment of tools to evaluate partnership outcomes and impacts, and reporting empirical psychometric evidence. Study, tool, psychometric and pragmatic characteristics were abstracted using a hybrid approach, then synthesized using descriptive statistics and thematic analysis. Study quality was assessed using the quality of survey studies in psychology (Q-SSP) checklist.

RESULTS

From 56 123 total citations, we screened 36 027 citations, assessed 2784 full-text papers, abstracted data from 48 studies and one companion report, and identified 58 tools. Most tools comprised surveys, questionnaires and scales. Studies used cross-sectional or mixed-method/embedded survey designs and employed quantitative and mixed methods. Both studies and tools were conceptually well grounded, focusing mainly on outcomes, then process, and less frequently on impact measurement. Multiple forms of empirical validity and reliability evidence was present for most tools; however, psychometric characteristics were inconsistently assessed and reported. We identified a subset of studies (22) and accompanying tools distinguished by their empirical psychometric, pragmatic and study quality characteristics. While our review demonstrated psychometric and pragmatic improvements over previous reviews, challenges related to health research partnership assessment and the nascency of partnership science persist.

CONCLUSION

This systematic review identified multiple tools demonstrating empirical psychometric evidence, pragmatic strength and moderate study quality. Increased attention to psychometric and pragmatic requirements in tool development, testing and reporting is key to advancing health research partnership assessment and partnership science. PROSPERO CRD42021137932.

How are health research partnerships assessed? A systematic review of outcomes, impacts, terminology and the use of theories, models and frameworks

BACKGROUND

Accurate, consistent assessment of outcomes and impacts is challenging in the health research partnerships domain. Increased focus on tool quality, including conceptual, psychometric and pragmatic characteristics, could improve the quantification, measurement and reporting partnership outcomes and impacts. This cascading review was undertaken as part of a coordinated, multicentre effort to identify, synthesize and assess a vast body of health research partnership literature.

OBJECTIVE

To systematically assess the outcomes and impacts of health research partnerships, relevant terminology and the type/use of theories, models and frameworks (TMF) arising from studies using partnership assessment tools with known conceptual, psychometric and pragmatic characteristics.

METHODS

Four electronic databases were searched (MEDLINE, Embase, CINAHL Plus and PsycINFO) from inception to 2 June 2021. We retained studies containing partnership evaluation tools with (1) conceptual foundations (reference to TMF), (2) empirical, quantitative psychometric evidence (evidence of validity and reliability, at minimum) and (3) one or more pragmatic characteristics. Outcomes, impacts, terminology, definitions and TMF type/use were abstracted verbatim from eligible studies using a hybrid (independent abstraction-validation) approach and synthesized using summary statistics (quantitative), inductive thematic analysis and deductive categories (qualitative). Methodological quality was assessed using the Quality Assessment Tool for Studies with Diverse Designs (QATSDD).

RESULTS

Application of inclusion criteria yielded 37 eligible studies. Study quality scores were high (mean 80%, standard deviation 0.11%) but revealed needed improvements (i.e. methodological, reporting, user involvement in research design). Only 14 (38%) studies reported 48 partnership outcomes and 55 impacts; most were positive effects (43, 90% and 47, 89%, respectively). Most outcomes were positive personal, functional, structural and contextual effects; most impacts were personal, functional and contextual in nature. Most terms described outcomes (39, 89%), and 30 of 44 outcomes/impacts terms were unique, but few were explicitly defined (9, 20%). Terms were complex and mixed on one or more dimensions (e.g. type, temporality, stage, perspective). Most studies made explicit use of study-related TMF (34, 92%). There were 138 unique TMF sources, and these informed tool construct type/choice and hypothesis testing in almost all cases (36, 97%).

CONCLUSION

This study synthesized partnership outcomes and impacts, deconstructed term complexities and evolved our understanding of TMF use in tool development, testing and refinement studies. Renewed attention to basic concepts is necessary to advance partnership measurement and research innovation in the field. Systematic review protocol registration: PROSPERO protocol registration: CRD42021137932 https://www.crd.york.ac.uk/prospero/display_record.php?RecordID=137932 .

Defining, conceptualizing and evaluating pragmatic qualities of quantitative instruments measuring implementation determinants and outcomes: a scoping and critical review of the literature and recommendations for future research

The pragmatic (i.e., practical) quality of quantitative implementation measures has received increased attention in the implementation science literature in recent years. Implementation measures that are judged to be pragmatic by implementation stakeholders are thought to be more likely to be applied in research and practice. Despite the need for pragmatic implementation measures, ambiguity and uncertainty regarding what constitutes a pragmatic measure remains. This study sought to identify and critically appraise the published literature to understand (i) how pragmatism is defined as a measurement construct/quality of implementation determinants and outcome instruments; (ii) how pragmatic qualities of instruments are evaluated; (iii) identify key gaps and limitations of the current evidence-base and (iv) identify recommendations for future research. We conducted a scoping review of the literature also employing methods of critical review. PubMed and PsycINFO databases, using the OVID interface, were searched for relevant articles published between January 2010 and September 2020. Articles that contained a definition and/or described characteristics of "pragmatism" as a measurement construct of quantitative implementation outcomes (as defined by Proctor's Implementation Outcomes taxonomy) and/or implementation determinants were eligible for inclusion. Nine articles met inclusion criteria. A degree of overlap in definitions and terms used to describe the pragmatic qualities of quantitative implementation determinant and outcome instruments were found. The most frequently cited descriptors of pragmatism were "not burdensome", "brief", "reliable", "valid" and "sensitive to change". 3 of the 9 included articles involved international implementation stakeholders in defining and conceptualizing pragmatism and employed specific methods to do so, including a systematic literature review, stakeholder interviews, concept mapping, and a Delphi process. All other articles defined pragmatism, with or without citing relevant literature. One article objectively assessed the pragmatic qualities, above and beyond the psychometric qualities, of implementation measures, using the Psychometric and Pragmatic Evidence Rating Scale (PAPERS). The evidence base within the implementation instrumentation literature on what pragmatism is and how it might be assessed is limited. Some of the research identified in the review provides a strong foundation to build upon, by testing its applicability in other settings (including healthcare areas and countries) and among a more diverse group of stakeholders. We discuss directions for further development of the concept of pragmatism relating to the measurement of implementation determinants and outcomes.

A Randomized Trial to Identify Accurate Measurement Methods for Adherence to Cognitive-Behavioral Therapy

Clinician fidelity to cognitive behavioral therapy (CBT) is an important mechanism by which desired clinical outcomes are achieved and is an indicator of care quality. Despite its importance, there are few fidelity measurement methods that are efficient and have demonstrated reliability and validity. Using a randomized trial design, we compared three methods of assessing CBT adherence-a core component of fidelity-to direct observation, the gold standard. Clinicians recruited from 27 community mental health agencies (n = 126; M age = 37.69 years, SD = 12.84; 75.7% female) were randomized 1:1:1 to one of three fidelity conditions: self-report (n = 41), chart-stimulated recall (semistructured interviews with the chart available; n = 42), or behavioral rehearsal (simulated role-plays; n = 43). All participating clinicians completed fidelity assessments for up to three sessions with three different clients that were recruited from clinicians' caseloads (n = 288; M age = 13.39 years SD = 3.89; 41.7% female); sessions were also audio-recorded and coded for comparison to determine the most accurate method. All fidelity measures had parallel scales that yielded an adherence maximum score (i.e., the highest-rated intervention in a session), a mean of techniques observed, and a count total of observed techniques. Results of three-level mixed effects regression models indicated that behavioral rehearsal produced comparable scores to observation for all adherence scores (all ps > .01), indicating no difference between behavioral rehearsal and observation. Self-report and chart-stimulated recall overestimated adherence compared to observation (ps < .01). Overall, findings suggested that behavioral rehearsal indexed CBT adherence comparably to direct observation, the gold-standard, in pediatric populations. Behavioral rehearsal may at times be able to replace the need for resource-intensive direct observation in implementation research and practice.

Elements for successful implementation of a clinic-based health literacy intervention

Improving health literacy is a national public health priority. Given the context of the COVID-19 pandemic, it is even more critical for health and medical information to be clear and understandable for patients and their families. Clinic-based programs to improve health literacy need to be pragmatic, feasible, and helpful for the implementing clinic and patients. This paper describes the development, implementation, and evaluation of a pragmatic, clinic-based health literacy intervention in a safety-net clinic that serves uninsured and indigent patients. Study methods are guided by a previous pilot study and components recommended for pragmatic interventions. An electronic readiness assessment was distributed to out-patient clinics affiliated with a statewide hospital association. The AskMe3 tool was used for the intervention as it is evidence informed and relatively easy to implement. Implementation included ongoing dialogue between the clinic and the academic research team. Within the implementing clinic, data collected from patients via verbally administered questionnaires was analyzed using descriptive statistics and chi-squares. Interview data collected from the clinic director was analyzed qualitatively for themes. The implementing clinic had some of the lowest average scores of the 34 clinics who participated in the initial readiness assessment. Despite this, they were able to successfully implement the health literacy intervention during a global pandemic. Eighty-eight participants completed patient questionnaires at this clinic. Most patients (96%) agreed the AskMe3 questions helped them talk with the doctor or nurse at their current appointment. Most (99%) also perceived the AskMe3 tool to be very helpful when used in a clinical setting. The clinic director offered that the staff initially thought the intervention would be difficult to implement. However, implementation by clinic volunteers with encouragement and prioritization of health literacy by the clinic director contributed to success. When considering interventions for clinical settings, a pragmatic approach can help with selection and implementation of a program that fits with the realities on the ground. Further, frequent technical assistance can help resolve implementation barriers. Interventions utilizing tools such as AskMe3, because of their simplicity, allow creative solutions to capacity issues for clinics who see a need for health literacy improvements.

The mechanics of implementation strategies and measures: advancing the study of implementation mechanisms

BACKGROUND

There is a fundamental gap in understanding the causal mechanisms by which strategies for implementing evidence-based practices address local barriers to effective, appropriate service delivery. Until this gap is addressed, scientific knowledge and practical guidance about which implementation strategies to use in which contexts will remain elusive. This research project aims to identify plausible strategy-mechanism linkages, develop causal models for mechanism evaluation, produce measures needed to evaluate such linkages, and make these models, methods, and measures available in a user-friendly website. The specific aims are as follows: (1) build a database of strategy-mechanism linkages and associated causal pathway diagrams, (2) develop psychometrically strong, pragmatic measures of mechanisms, and (3) develop and disseminate a website of implementation mechanisms knowledge for use by diverse stakeholders.

METHODS

For the first aim, a combination of qualitative inquiry, expert panel methods, and causal pathway diagramming will be used to identify and confirm plausible strategy-mechanism linkages and articulate moderators, preconditions, and proximal and distal outcomes associated with those linkages. For the second aim, rapid-cycle measure development and testing methods will be employed to create reliable, valid, pragmatic measures of six mechanisms of common strategies for which no high-quality measures exist. For the third aim, we will develop a user-friendly website and searchable database that incorporates user-centered design, disseminating the final product using social marketing principles.

DISCUSSION

Once strategy-mechanism linkages are identified using this multi-method approach, implementation scientists can use the searchable database to develop tailored implementation strategies and generate more robust evidence about which strategies work best in which contexts. Moreover, practitioners will be better able to select implementation strategies to address their specific implementation problems. New horizons in implementation strategy development, optimization, evaluation, and deployment are expected to be more attainable as a result of this research, which will lead to enhanced implementation of evidence-based interventions for cancer control, and ultimately improvements in patient outcomes.

Quantitative Measurements for Factors Influencing Implementation in School Settings: Protocol for A Systematic Review and A Psychometric and Pragmatic Analysis

INTRODUCTION

In order to address the effectiveness and sustainability of school-based interventions, there is a need to consider the factors affecting implementation success. The rapidly growing field of implementation-focused research is struggling to determine how to assess and measure implementation-relevant constructs. Earlier research has identified the need for strong psychometric and pragmatic measures. The aims of this review are therefore to (i) systematically review the literature to identify measurements of the factors influencing implementations which have been developed or adapted in school settings, (ii) describe each measurement's psychometric and pragmatic properties, (iii) describe the alignment between each measurement and the corresponding domain and/or construct of the Consolidated Framework for Implementation Research (CFIR).

METHODS

Six databases (Medline, ERIC, PsycInfo, Cinahl, Embase, and Web of Science) will be searched for peer-reviewed articles reporting on school settings, published from the year 2000. The identified measurements will be mapped against the CFIR, and analyzed for their psychometric and pragmatic properties.

DISCUSSION

By identifying measurements that are psychometrically and pragmatically impactful in the field, this review will contribute to the identification of feasible, effective, and sustainable implementation strategies for future research in school settings.

Repetitive Behavior Scale for Early Childhood (RBS-EC): Psychometrics and Developmental Effects with a Community Sample

Restricted, repetitive behaviors and interests (RRBIs) occur commonly in young children in the course of typical development, but too frequent or persistent RRBIs have been linked to a variety of psychosocial and neurodevelopmental disorders. Unfortunately, a lack of psychometrically sound, feasible instruments stymies RRBI study and intervention in early childhood. Therefore, this study validated an extant RRBI measure (i.e., Repetitive Behavior Scale for Early Childhood; RBS-EC) for predominately neurotypical community children ages 2-7 years via an MTurk survey of 300 caregivers. Confirmatory factor analyses best supported a 4-factor model for the RBS-EC (i.e., repetitive motor, rituals and routines, restricted interests, self-directed behaviors). RBS-EC scores varied significantly with age and gender; namely, RRBIs were typically (1) less frequent but more problematic with older versus younger children and (2) more frequent and problematic with boys versus girls. Findings extend the RBS-EC's utility in RRBI assessment and treatment during early childhood.

Intervention Activities Associated with the Implementation of a Comprehensive School Tobacco Policy at Danish Vocational Schools: A Repeated Cross-Sectional Study

School tobacco policies are often poorly implemented, which may explain their limited effectiveness. Further, constructs to measure implementation outcomes of school tobacco policies are missing. The Smoke-Free Vocational Schools intervention was designed to stimulate the implementation of a comprehensive school tobacco policy into routine practice. This study (1) developed implementation fidelity outcomes measures for the school tobacco policy and (2) examined associations between intervention activities and implementation fidelity at two time points. We applied a repeated cross-sectional survey study design across seven schools: the first time point was >5 months after the policy was established and the second time point > 14 months after policy establishment. The dependent/outcome variables were four binary fidelity domains as well as a total score across domains. A total of six intervention activities were measured among either students (e.g., new school-break facilities) or staff/managers (e.g., a joint workshop before policy implementation). Associations were analyzed separately for students and staff/managers using generalized linear mixed models, adjusted for confounders. A total of n = 2674 students and n = 871 staff/managers participated. The total implementation fidelity scores increased over time among both students and staff/managers. Three intervention activities were consistently associated with the total implementation fidelity score, including: new school-break facilities (B_T1 = 0.08, 95% CI = 0.03; 0.12; B_T2 = 0.07, 95% CI = 0.04-0.50), the joint workshop before policy implementation (B_T1 = 0.13, 95% CI = 0.02; 0.25; B_T2 = 0.13, 95% CI = 0.004; 0.24), and internalization of fixed procedures for enforcement (B_T1 = 0.19, 95% CI = 0.13-0.26; B_T2 = 0.16, 95% CI = 0.13-0.26). These findings can be applied by schools and other actors in practice. The developed implementation fidelity outcomes measures can be applied in future research on school tobacco policies.

The impact of an autonomous nurse-led high-flow nasal cannula oxygen protocol on clinical outcomes of infants with bronchiolitis

OBJECTIVE

To investigate the relationship of the implementation of a nurse-led high-flow nasal cannula oxygen protocol on the clinical outcomes of infants with bronchiolitis in a regional paediatric unit.

BACKGROUND

Bronchiolitis is a common lower respiratory illness and is the leading cause for hospitalisation of infants globally. Standard care involves the provision of supportive measures. Historically, supplemental oxygen was provided by low-flow nasal cannula. High-flow nasal cannula oxygen has been increasingly adopted despite limited evidence of its efficacy.

METHODS

This study employed non-equivalent, post-implementation only design to explore clinical outcomes of infants with bronchiolitis admitted for high-flow nasal cannula oxygen therapy. The study compared infants in the 24 months before and after the initiation of a high-flow nasal cannula protocol. The primary clinical outcome was length of stay, secondary outcomes included time on high flow, weaning time, escalation of care and time outside of physiological parameters. Implementation strategy evaluation was measured by compliance with applying the protocol, reported as episodes of variance, and duration of variance. The StaRI checklist was selected as the most appropriate reporting guideline.

RESULTS

A total of 80 patients were admitted with bronchiolitis and received high-flow nasal cannula oxygen therapy during a 48-month period; 37 patients were prior, and 43 after, the introduction of a nurse-led high-flow nasal cannula protocol. Length of stay was significantly reduced in the post-implementation group compared to the historical control group (83.8 vs. 61.3 h). Time on high flow and weaning time was decreased in the post-implementation group compared to the control group (33.5 vs. 26.7 h and 26 vs.12.25 h, respectively); however, these did not reach statistical significance. There was varied application of the HFNC protocol.

CONCLUSIONS

The implementation of a nurse-led high-flow nasal cannula protocol was associated with a reduced length of stay.

RELEVANCE TO CLINICAL PRACTICE

This study demonstrated that infants with bronchiolitis that were treated with a nurse-led high-flow nasal cannula (HFNC) therapy protocol had positive effects on clinical outcomes including a shorter length of stay than compared with those with physician-directed care in a regional paediatric unit. A weight-based (2 L/kg) HFNC therapy was safely administered to infants with bronchiolitis in a regional hospital paediatric ward with no paediatric intensive care unit (PICU).

A citation analysis and scoping systematic review of the operationalization of the Practical, Robust Implementation and Sustainability Model (PRISM)

BACKGROUND

The Practical, Robust Implementation and Sustainability Model (PRISM) was developed in 2008 as a contextually expanded version of the broadly used Reach, Adoption, Effectiveness, Implementation, and Maintenance (RE-AIM) framework. PRISM provides researchers a pragmatic and intuitive model to improve translation of research interventions into clinical and community practice. Since 2008, the use of PRISM increased across diverse topics, populations, and settings. This citation analysis and scoping systematic review aimed to assess the use of the PRISM framework and to make recommendations for future research.

METHODS

A literature search was conducted using three databases (PubMed, Web of Science, Scopus) for the period of 2008 and September 2020. After exclusion, reverse citation searches and invitations to experts in the field were used to identify and obtain recommendations for additional articles not identified in the original search. Studies that integrated PRISM into their study design were selected for full abstraction. Unique research studies were abstracted for information on study characteristics (e.g., setting/population, design), PRISM contextual domains, and RE-AIM outcomes.

RESULTS

A total of 180 articles were identified to include PRISM to some degree. Thirty-two articles representing 23 unique studies integrated PRISM within their study design. Study characteristics varied widely and included studies conducted in diverse contexts, but predominately in high-income countries and in clinical out-patient settings. With regards to use, 19 used PRISM for evaluation, 10 for planning/development, 10 for implementation, four for sustainment, and one for dissemination. There was substantial variation across studies in how and to what degree PRISM contextual domains and RE-AIM outcomes were operationalized and connected. Only two studies directly connected individual PRISM context domains with RE-AIM outcomes, and another four included RE-AIM outcomes without direct connection to PRISM domains.

CONCLUSIONS

This is the first systematic review of the use of PRISM in various contexts. While there were low levels of 'integrated' use of PRISM and few reports on linkage to RE-AIM outcomes, most studies included important context domains of implementation and sustainability infrastructure and external environment. Recommendations are provided for more consistent and comprehensive use of and reporting on PRISM to inform both research and practice on contextual factors in implementation.

Research and Scholarly Methods: Implementation Science Studies

Traditional research focuses on efficacy or effectiveness of interventions but lacks evaluation of strategies needed for equitable uptake, scalable implementation, and sustainable evidence-based practice transformation. The purpose of this introductory review is to describe key implementation science (IS) concepts as they apply to medication management and pharmacy practice, and to provide guidance on literature review with an IS lens. There are five key ingredients of IS, including: (1) evidence-based intervention; (2) implementation strategies; (3) IS theory, model, or framework; (4) IS outcomes and measures; and (5) stakeholder engagement, which is key to a successful implementation. These key ingredients apply across the three stages of IS research: (1) pre-implementation; (2) implementation; and (3) sustainment. A case example using a combination of IS models, PRISM (Practical, Robust Implementation and Sustainability model) and RE-AIM (Reach, Effectiveness, Adoption, Implementation, Maintenance), is included to describe how an IS study is designed and conducted. This case is a cluster randomized trial comparing two clinical decision support tools to improve guideline-concordant prescribing for patients with heart failure and reduced ejection fraction. The review also includes information on the Standards for Reporting Implementation Studies (StaRI), which is used for literature review and reporting of IS studies,as well as IS-related learning resources.

Promises and pitfalls in implementation science from the perspective of US-based researchers: learning from a pre-mortem

BACKGROUND

Implementation science is at a sufficiently advanced stage that it is appropriate for the field to reflect on progress thus far in achieving its vision, with a goal of charting a path forward. In this debate, we offer such reflections and report on potential threats that might stymie progress, as well as opportunities to enhance the success and impact of the field, from the perspective of a group of US-based researchers.

MAIN BODY

Ten mid-career extramurally funded US-based researchers completed a "pre-mortem" or a group brainstorming exercise that leverages prospective hindsight to imagine that an event has already occurred and to generate an explanation for it - to reduce the likelihood of a poor outcome. We came to consensus on six key themes related to threats and opportunities for the field: (1) insufficient impact, (2) too much emphasis on being a "legitimate science," (3) re-creation of the evidence-to-practice gap, (4) difficulty balancing accessibility and field coherence, (5) inability to align timelines and priorities with partners, and (6) overly complex implementation strategies and approaches.

CONCLUSION

We submit this debate piece to generate further discussion with other implementation partners as our field continues to develop and evolve. We hope the key opportunities identified will enhance the future of implementation research in the USA and spark discussion across international groups. We will continue to learn with humility about how best to implement with the goal of achieving equitable population health impact at scale.

Lessons learned from two clinical trials on nutritional supplements to reduce aggressive behaviour

BACKGROUND

Setting up and conducting a randomised controlled trial (RCT) has many challenges-particularly trials that include vulnerable individuals with behavioural problems or who reside in facilities that focus on care as opposed to research. These populations are underrepresented in RCTs.

APPROACH

In our paper, we describe the challenges and practical lessons learned from two RCTs in two care settings involving long-stay psychiatric inpatients and people with intellectual disabilities. We describe five main difficulties and how these were overcome: (1) multisite setting, (2) inclusion of vulnerable participants, (3) nutritional supplements and placebos, (4) assessment of behavioural outcomes, and (5) collecting bio samples.

CONCLUSIONS

By sharing these practical experiences, we hope to inform other researchers how to optimally design their trials, while avoiding and minimising the difficulties that we encountered, and to facilitate the implementation of a trial. Both trials were registered in the Clinical Trials Register (RCT A: NCT02498106; RCT B: NCT03212092).

Methods to improve the translation of evidence-based interventions: A primer on dissemination and implementation science for occupational safety and health researchers and practitioners

Objective

A limited focus on dissemination and implementation (D&I) science has hindered the uptake of evidence-based interventions (EBIs) that reduce workplace morbidity and mortality. D&I science methods can be used in the occupational safety and health (OSH) field to advance the adoption, implementation, and sustainment of EBIs for complex workplaces. These approaches should be responsive to contextual factors, including the needs of partners and beneficiaries (such as employers, employees, and intermediaries).

Methods

By synthesizing seminal literature and texts and leveraging our collective knowledge as D&I science and/or OSH researchers, we developed a D&I science primer for OSH. First, we provide an overview of common D&I terminology and concepts. Second, we describe several key and evolving issues in D&I science: balancing adaptation with intervention fidelity and specifying implementation outcomes and strategies. Next, we review D&I theories, models, and frameworks and offer examples for applying these to OSH research. We also discuss widely used D&I research designs, methods, and measures. Finally, we discuss future directions for D&I science application to OSH and provide resources for further exploration.

Results

We compiled a D&I science primer for OSH appropriate for practitioners and evaluators, especially those newer to the field.

Conclusion

This article fills a gap in the OSH research by providing an overview of D&I science to enhance understanding of key concepts, issues, models, designs, methods and measures for the translation into practice of effective OSH interventions to advance the safety, health and well-being of workers.

Leading the charge in the education sector: development and validation of the School Implementation Leadership Scale (SILS)

Background

Strategic implementation leadership is a critical determinant of successful implementation, hypothesized to create a more supportive implementation climate conducive to the adoption and use of evidence-based practices. Implementation leadership behaviors may vary significantly across contexts, necessitating studies that examine the validity of established measurement tools in novel health service delivery sectors. The education sector is the most common site for delivering mental health services to children and adolescents in the USA, but research focused on implementation leadership in schools is in the early phases, and there is a need for adaptation and expansion of instruments in order to tailor to the school context. The current study adapted and validated the School Implementation Leadership Scale (SILS) (based on the Implementation Leadership Scale) in a sample of elementary school personnel from six school districts who were implementing one of two well-established prevention programs for supporting children’s mental health.

Methods

Participants were 441 public school teachers from 52 elementary schools in the Midwest and West Coast of the USA. Participants completed a survey that contained: (1) an adapted and expanded version of the SILS with additional items generated for four existing subscales as well as three new subscales (communication, vision/mission, and availability), and (2) additional tools to evaluate convergent and divergent validity (i.e., measures of general/molar leadership and teaching attitudes). Data underwent (1) examination of item characteristic curves to reduce items and ensure a pragmatic instrument, (2) confirmatory factor analyses to establish structural validity, and (3) evaluation of convergent and divergent validity.

Results

Item reduction analyses resulted in seven subscales of three items each. Results indicated acceptable fit for a seven-factor structural model (CFI = .995, TLI = .99, RMSEA = .07, SRMR = 0.02). Second-order factor loadings were high (λ = .89 to .96), suggesting that the SILS subscales comprise a higher-order implementation leadership factor. All subscales demonstrated good inter-item reliability (α = .91–.96). Convergent and divergent validity results were generally as hypothesized, with moderate to high correlations between SILS subscales and general leadership, moderate correlations with teaching attitudes, and low correlations with school demographics.

Conclusions

Overall, results provided strong structural, convergent, and divergent validity evidence for the 21-item, 7-factor SILS instrument. Implications for the measurement of implementation leadership in schools are discussed, as well as strategies to support leaders to enhance their strategic behaviors related to the implementation of mental health prevention programs (e.g., adaptation of existing leadership-focused implementation strategies).

Supplementary Information

The online version contains supplementary material available at 10.1186/s13012-022-01222-7.

Teacher Perceptions of Implementation Climate Related to Feasibility of Implementing Schoolwide Positive Behavior Supports and Interventions

For Schoolwide Positive Behavior Interventions and Supports (SWPBIS) to be effective, educators must use the practices as intended. Whether a teacher uses a practice as intended can depend, in part, on how feasible the practice is perceived to be. Identifying malleable factors associated with teachers’ perceptions of SWPBIS feasibility can help schools identify targeted supports to improve feasibility. Implementation climate, or the shared perception among implementers that their school supports implementation efforts, is known to promote high quality implementation. However, little is known about how individuals’ perceptions of their school’s implementation climate may influence their perceptions of feasibility. The lack of empirical evidence points to a need to explore whether educators’ shared and individual perceptions of implementation climate influence feasibility of implementing SWPBIS. Therefore, this study examines the link between teachers’ individual and shared perceptions of implementation climate related to the feasibility of implementing SWPBIS in a sample of 348 K-5 general education teachers across 39 elementary schools in the pacific northwestern United States. Results indicate that teachers who experience their schools’ implementation climate as positive are more likely to report SWPBIS as feasible, controlling for teachers’ shared perceptions of implementation climate. Implications for schools aiming to improve their implementation of SWPBIS, including the development of individualized implementation supports (e.g., tailoring implementation strategies to support each and every teacher), are discussed.

Development and Feasibility Testing of the Clinical-Community Linkage Self-Assessment Survey for Community Organizations

Introduction

Clinical-community linkages (CCLs) can improve health, but few instruments exist to evaluate these partnerships. To address this gap, we develop and test the Clinical-Community Linkage Self-Assessment Survey (CCL Self-Assessment).

Materials and Methods

We built on an existing framework and conducted a literature review to guide the design of our survey, and obtained feedback from academic, clinical, and community-based experts. To pretest the instrument, we conducted 10 think-aloud interviews with community-based health-promotion organizations. We performed feasibility testing with 38 staff from 20 community organizations, followed by criterion-validity testing.

Results

The 15-item final instrument includes five domains: Nature of the Relationship, Communication, Referral Process, Feedback Loop, and Timeliness. Expert feedback included keeping the CCL Self-Assessment brief and actionable. Think-aloud interviews produced a range of revisions related to item wording, instructions, brevity, and formatting. Feasibility testing showed high response rate and ease of administration. Sites scoring high on the CCL Self-Assessment also scored high on the criterion measure.

Discussion

We demonstrate feasibility, as well as face, content, construct, and criterion validity. Initial results suggest the CCL Self-Assessment survey may be used by community organizations to identify strengths and weaknesses of their linkages. Next steps include additional statistical validation and testing to determine how the CCL Self-Assessment survey works in the field as well as providing specific tools to improve linkages.

Psychometric performance of the Mental Health Implementation Science Tools (mhIST) across six low- and middle-income countries

BACKGROUND

Existing implementation measures developed in high-income countries may have limited appropriateness for use within low- and middle-income countries (LMIC). In response, researchers at Johns Hopkins University began developing the Mental Health Implementation Science Tools (mhIST) in 2013 to assess priority implementation determinants and outcomes across four key stakeholder groups-consumers, providers, organization leaders, and policy makers-with dedicated versions of scales for each group. These were field tested and refined in several contexts, and criterion validity was established in Ukraine. The Consumer and Provider mhIST have since grown in popularity in mental health research, outpacing psychometric evaluation. Our objective was to establish the cross-context psychometric properties of these versions and inform future revisions.

METHODS

We compiled secondary data from seven studies across six LMIC-Colombia, Myanmar, Pakistan, Thailand, Ukraine, and Zambia-to evaluate the psychometric performance of the Consumer and Provider mhIST. We used exploratory factor analysis to identify dimensionality, factor structure, and item loadings for each scale within each stakeholder version. We also used alignment analysis (i.e., multi-group confirmatory factor analysis) to estimate measurement invariance and differential item functioning of the Consumer scales across the six countries.

RESULTS

All but one scale within the Provider and Consumer versions had Cronbach's alpha greater than 0.8. Exploratory factor analysis indicated most scales were multidimensional, with factors generally aligning with a priori subscales for the Provider version; the Consumer version has no predefined subscales. Alignment analysis of the Consumer mhIST indicated a range of measurement invariance for scales across settings (R2 0.46 to 0.77). Several items were identified for potential revision due to participant nonresponse or low or cross- factor loadings. We found only one item, which asked consumers whether their intervention provider was available when needed, to have differential item functioning in both intercept and loading.

CONCLUSION

We provide evidence that the Consumer and Provider versions of the mhIST are internally valid and reliable across diverse contexts and stakeholder groups for mental health research in LMIC. We recommend the instrument be revised based on these analyses and future research examine instrument utility by linking measurement to other outcomes of interest.

The Stability of Nursing Home Residents' Ratings of Importance of Recreation Preferences Over One Year

Long-term stability of nursing home (NH) residents' everyday preference remains unknown. We examined 1-year stability in reports of importance of 34-recreational activity preferences (8-MDS 3.0 Section F items; 26-Preferences for Everyday Living Inventory-NH items) by NH residents (N = 161). We examined mean differences on demographic and clinical characteristics of residents for preferences showing change. Importance ratings of preferences were highly stable over 1-year, with 91% of items retaining the same valence of importance for the majority of the sample (<20% change). Three preferences showed greater change. More functionally able residents were more likely to change their importance on "being with groups of people," and older residents were more likely to change their preferences for being "involved in religious practices" and "around animals such as pets". Overall, annual assessments of recreational activity preferences capture an accurate representation of preferences with reassessment only needed in a few circumstances.

Don't Get Lost in Translation: Integrating Developmental and Implementation Sciences to Accelerate Real-World Impact on Children's Development, Health, and Wellbeing

Translation of developmental science discoveries is impeded by numerous barriers at different stages of the research-to-practice pipeline. Actualization of the vast potential of the developmental sciences to improve children's health and development in the real world is imperative but has not yet been fully realized. In this commentary, we argue that an integrated developmental-implementation sciences framework will result in a translational mindset essential for accelerating real world impact. We delineate key principles and methods of implementation science of salience to the developmental science audience, lay out a potential synthesis between implementation and developmental sciences, provide an illustration of the Mental Health, Earlier Partnership (MHE-P), and set actionable steps for realization. Blending these approaches along with wide-spread adoption of the translational mindset has transformative potential for population-level impact of developmental science discovery.