Validity Evidence for an Observational Fidelity Measure to Inform Scale-Up of Evidence-Based Interventions

As evidence-based interventions are scaled, fidelity of implementation, and thus effectiveness, often wanes. Validated fidelity measures can improve researchers' ability to attribute outcomes to the intervention and help practitioners feel more confident in implementing the intervention as intended. We aim to provide a model for the validation of fidelity observation protocols to guide future research studying evidence-based interventions scaled-up under real-world conditions. We describe a process to build evidence of validity for items within the Session Review Form, an observational tool measuring fidelity to interactive drug prevention programs such as the Botvin LifeSkills Training program. Following Kane's (2006) assumptions framework requiring that validity evidence be built across four areas (scoring, generalizability, extrapolation, and decision), confirmatory factor analysis supported the hypothesized two-factor structure measuring quality of delivery (seven items assessing how well the material is implemented) and participant responsiveness (three items evaluating how well the intervention is received), and measurement invariance tests suggested the structure held across grade level and schools serving different student populations. These findings provide some evidence supporting the extrapolation assumption, though additional research is warranted since a more complete overall depiction of the validity argument is needed to evaluate fidelity measures.

Racial and Ethnic Representation in Preventive Intervention Research: a Methodological Study

Individuals who are Asian or Asian American, Black or African American, Native American or American Indian or Alaska Native, Native Hawaiian or Pacific Islander, and Hispanic or Latino (i.e., presently considered racial ethnic minoritized groups in the USA) lacked equal access to resources for mitigating risk during COVID-19, which highlighted public health disparities and exacerbated inequities rooted in structural racism that have contributed to many injustices, such as failing public school systems and unsafe neighborhoods. Minoritized groups are also vulnerable to climate change wherein the most severe harms disproportionately fall upon underserved communities. While systemic changes are needed to address these pervasive syndemic conditions, immediate efforts involve examining strategies to promote equitable health and well-being-which served as the impetus for this study. We conducted a descriptive analysis on the prevalence of culturally tailored interventions and reporting of sample characteristics among 885 programs with evaluations published from 2010 to 2021 and recorded in the Blueprints for Healthy Youth Development registry. Inferential analyses also examined (1) reporting time trends and (2) the relationship between study quality (i.e., strong methods, beneficial effects) and culturally tailored programs and racial ethnic enrollment. Two percent of programs were developed for Black or African American youth, and 4% targeted Hispanic or Latino populations. For the 77% of studies that reported race, most enrollees were White (35%) followed by Black or African American (28%), and 31% collapsed across race or categorized race with ethnicity. In the 64% of studies that reported ethnicity, 32% of enrollees were Hispanic or Latino. Reporting has not improved, and there was no relationship between high-quality studies and programs developed for racial ethnic youth, or samples with high proportions of racial ethnic enrollees. Research gaps on racial ethnic groups call for clear reporting and better representation to reduce disparities and improve the utility of interventions.

Improving the livelihoods of justice-involved youth: Study protocol for a replication and extension of functional family therapy-gangs

BACKGROUND

This study will evaluate Functional Family Therapy-Gangs (FFT-G), an extension of a family-based therapeutic intervention-Functional Family Therapy (FFT)-designed to help troubled youth exhibiting mild to severe behavior problems overcome delinquency, substance abuse, and violence. FFT-G, however, addresses risk factors that are typically more salient among gang than delinquent populations. A randomized control trial with adjudicated youth in Philadelphia revealed reductions in recidivism over an 18-month period. The purposes of this paper are to outline the protocol for replicating FFT-G in the Denver metropolitan area, document the design and challenges of the prospective research, and promote transparency.

METHODS

As a condition of pre-trial or probation supervision, 400 youth/caregiver dyads will be randomly assigned to FFT-G or a treatment-as-usual control group. Preregistered confirmatory outcomes include recidivism (i.e., criminal/delinquent charges and adjudications/convictions) measured using official records (Open Science Framework: https://osf.io/abyfs). Secondary outcomes include measures of gang embeddedness, non-violent and violent recidivism, and substance use measured using interview-based surveys and arrest, revocation, incarceration, and crime type indicators of recidivism from official records. Exploratory mediation and moderation analyses are also planned. Intent-to-treat regression analyses will estimate intervention effects 18 months post-randomization.

CONCLUSION

This study will contribute to advancing high-quality evidence-based knowledge on gang interventions for which there are few known effective responses.

The influence of evidence-based program registry websites for dissemination of evidence-based interventions in behavioral healthcare

PURPOSE

Evidence-based program registries (EBPRs) are web-based databases of evaluation studies that summarize the available evidence for the effectiveness of behavioral healthcare programs, including programs addressing substance misuse, mental health, child welfare, or offender rehabilitation. The study determined the extent to which visitors to selected EBPRs accomplished the objectives of their visits and how often those visits resulted in the adoption of new or improved evidence-based interventions (EBIs).

METHOD

A follow-up telephone survey was conducted with 216 visitors to a convenience sample of six EBPRs an average of six months after the visitors' incident visit to the EBPR.

RESULTS

The most frequent objective was to identify evidence-based programs/services, curricula or assessments, followed by finding resources to implement or improve the preceding and writing a grant proposal including to comply with funding requirements; 71% of such objectives were achieved across the full set of objectives. Implementation of an EBI was completely achieved for 31% of relevant objectives and some progress on EBI implementation occurred for 19% of relevant objectives.

CONCLUSIONS

This is the first study to document the usage of EBPRs as a modality to increase the utilization of EBIs in the actual practice of behavioral healthcare. The results support the continued use of web-based EBPRs for disseminating information on evidence-based interventions for behavioral healthcare.

Substance use prevention during adolescence: Study protocol for a large-scale cluster randomized trial of Botvin High School LifeSkills Training

BACKGROUND

Concerns about the rise in adolescent vaping and cannabis use suggest the need for effective substance use prevention programs. Botvin LifeSkills Training (LST) has a strong evidence base at the middle school level for preventing or reducing tobacco use and related problems. A high school (grades 9-10) version of the LST program was also developed and shows promising initial evidence for reducing tobacco use in a single pilot study. However, the high school version of LST has not been sufficiently tested in an experimental trial, despite being widely implemented in high schools across the U.S. This paper outlines the study protocol for a large-scale cluster randomized trial of Botvin High School LST, with objectives of documenting the design of prospective research and promoting transparency.

METHODS

A total of 60 high schools in Colorado and Ohio were randomized to the 10-session, teacher-led intervention group (n = 33 schools) or business-as-usual control group (n = 27 schools). Across two cohorts of schools, 9th-grade students complete self-report surveys at pretest, immediate posttest, 1-year follow-up, and 21-month follow-up. Primary outcomes are tobacco (nicotine) use and cannabis use. Secondary outcomes are alcohol use, illicit drug use, psychosocial behaviors (e.g., violence and mental health), and academic achievement. Intent-to-treat analyses will use multilevel modeling to estimate intervention effects across assessment points.

CONCLUSION

This independent evaluation will help to determine whether the intervention is appropriate for large-scale adoption. This trial is preregistered with the Open Science Framework (https://osf.io/dnz5q/).

Visitors' Assessment and Utilization of Evidence-Based Program Resources (EBPR) Websites

Background: Evidence-based program resources (EBPR) websites for behavioral health are a potentially useful tool to assist decision-makers and practitioners in deciding which behavioral health interventions to implement. EBPR websites apply rigorous research standards to assess the effectiveness of behavioral healthcare programs, models, and clinical practices. Method: Visitors to a convenience sample of six EBPR websites (N=369, excluding students) were recruited for telephone interviews primarily by means of a pop-up invitation on the sites. Results: The visitors view the EBPR sites as important sources of information to support the identification and adoption of evidence-based programs/practices (EBPs) in behavioral healthcare, which aligns with the primary mission of EBPRs. For repeat visitors, there was some indication that the information obtained helped effect certain changes in their agencies' programs and policies. However, increased or improved guidance on EBP implementation was also requested. Conclusion: EBPR websites should be better publicized to the behavioral healthcare field.

Common Methodological Problems in Randomized Controlled Trials of Preventive Interventions

Randomized controlled trials (RCTs) are often considered the gold standard in evaluating whether intervention results are in line with causal claims of beneficial effects. However, given that poor design and incorrect analysis may lead to biased outcomes, simply employing an RCT is not enough to say an intervention "works." This paper applies a subset of the Society for Prevention Research (SPR) Standards of Evidence for Efficacy, Effectiveness, and Scale-up Research, with a focus on internal validity (making causal inferences) to determine the degree to which RCTs of preventive interventions are well-designed and analyzed, and whether authors provide a clear description of the methods used to report their study findings. We conducted a descriptive analysis of 851 RCTs published from 2010 to 2020 and reviewed by the Blueprints for Healthy Youth Development web-based registry of scientifically proven and scalable interventions. We used Blueprints' evaluation criteria that correspond to a subset of SPR's standards of evidence. Only 22% of the sample satisfied important criteria for minimizing biases that threaten internal validity. Overall, we identified an average of 1-2 methodological weaknesses per RCT. The most frequent sources of bias were problems related to baseline non-equivalence (i.e., differences between conditions at randomization) or differential attrition (i.e., differences between completers versus attritors or differences between study conditions that may compromise the randomization). Additionally, over half the sample (51%) had missing or incomplete tests to rule out these potential sources of bias. Most preventive intervention RCTs need improvement in rigor to permit causal inference claims that an intervention is effective. Researchers also must improve reporting of methods and results to fully assess methodological quality. These advancements will increase the usefulness of preventive interventions by ensuring the credibility and usability of RCT findings.

The Role of Clearinghouses in Promoting Transparent Research: A Methodological Study of Transparency Practices for Preventive Interventions

Transparency of research methods is vital to science, though incentives are variable, with only some journals and funders adopting transparency policies. Clearinghouses are also important stakeholders; however, to date none have implemented formal procedures that facilitate transparent research. Using data from the longest standing clearinghouse, we examine transparency practices for preventive interventions to explore the role of online clearinghouses in incentivizing researchers to make their research more transparent. We conducted a descriptive analysis of 88 evaluation reports reviewed in 2018-2019 by Blueprints for Healthy Youth Development, when the clearinghouse began checking for trial registrations, and expanded on these efforts by applying broader transparency standards to interventions eligible for an endorsement on the Blueprints website during the study period. Reports were recent, with 84% published between 2010 and 2019. We found that few reports had data, code, or research materials that were publicly available. Meanwhile, 40% had protocols that were registered, but only 8% were registered prospectively, while one-quarter were registered before conducting analyses. About one-third included details in a registered protocol describing the treatment contrast and planned inclusions, and less than 5% had a registered statistical analysis plan (e.g., planned analytical methods, pre-specified covariates). Confirmatory research was distinguished from exploratory work in roughly 40% of reports. Reports published more recently (after 2015) had higher rates of transparency. Preventive intervention research needs to be more transparent. Since clearinghouses rely on robust findings to make well-informed decisions and researchers are incentivized to meet clearinghouse standards, clearinghouses should consider policies that encourage transparency to improve the credibility of evidence-based interventions.

Making Evidence-Based Interventions Relevant for Users: A Comparison of Requirements for Dissemination Readiness Across Program Registries

This study compares prevention program registries in current use on their level of support for users seeking to implement evidence-based programs. Despite the importance of registries as intermediaries between researchers and the public, and although previous studies have examined how registries define their standards for methodological soundness and evidence of efficacy, little research has focused on the degree to which registries consider programs' dissemination readiness. The result is that registry users are uncertain whether listed programs and their necessary support materials are even available for implementation. This study evaluates 11 publicly and privately funded prevention registries that review the evidence base of programs seeking to improve child health and prosocial outcomes on the degree to which they use dissemination readiness as an evidentiary criterion for rating programs, and the extent and type of information they provide about dissemination readiness to support real-world implementation. The results show wide variability, with few having standards about dissemination readiness or making evidence-based information about interventions easily accessible to users. Findings indicate the need for registries to (1) do more to assess dissemination readiness before including programs on their website and (2) offer more complete information on dissemination readiness and implementation support to users.

Embolectomy in a rabbit acute arterial occlusion model using a novel electromechanical extraction device

A prototype endovascular electromechanical clot-extraction device was fabricated using a combination of shape memory polymer and shape memory nickel-titanium alloy (nitinol). Five embolic vascular occlusions were created in 4 rabbits by injecting thermally coagulated blood through a 4F catheter in the common carotid artery. Angiography immediately after clot injection showed complete or partial occlusion of the common carotid artery. Posttreatment angiography showed complete (2/5), partial (2/5), or no (1/5) restoration of blood flow.

The relationship between mothers' child abuse potential and current Mental Health symptoms

This analysis examined data from mothers at 2 of the 9 sites participating in Substance Abuse and Mental Health Services Administration's (SAMHSA's) national Women Co-occurring Disorders and Violence Study (WCDVS). According to previous literature, it was hypothesized that women in the WCDVS would be at high risk of perpetrating child abuse. This research examined mothers' potential for physical child abuse and assessed the association between child abuse potential, current mental health symptoms, alcohol and drug use severity, and trauma. Results revealed that participants had significant potential for child abuse. Hierarchical regression analyses revealed that current mental health symptoms were the strongest predictor of mothers' scores on the Child Abuse Potential (CAP) Inventory. This study highlights the important relationships between commonly used instruments across the mental health, substance, and child welfare fields and the potential dual use of these instruments. Implications for policy and practice are discussed.

Measles in the 1980s

We detail aspects of measles immunization programmes in several countries. Live measles vaccine has been available in Australia for 16 years, yet, in 1981, there were outbreaks of measles in the State of New South Wales (population 5 200 000) which led to 2200 admissions to hospital and five deaths. In response to complaints of "vaccine failure", a survey determined that 22.5% of children with measles seen by general practitioners and 10.3% of those admitted to hospitals had been previously immunized. There was no evidence of waning immunity, and noparticular batch of vaccine was implicated. The vaccine failures are attributed in part to failure of seroconversion in some recipients when immunized at 12 months of age as a result of interference by transplacentally acquired antibodies. As more of the susceptible population is vaccinated, there will be fewer cases of measles, but among these cases will be an increasing proportion of cases occurring in previously vaccinated individuals. The equation to calculate this expected proportion of "vaccine failures" is given. We support the measures to increase immunization compliance.