Promoting Learning from Null or Negative Results in Prevention Science Trials

Effectiveness of an intervention to improve ART adherence among men who have sex with men living with HIV: a randomized controlled trial in three public HIV clinics in Mexico

We conducted a parallel-group randomized controlled trial in three HIV clinics in Mexico to evaluate a user-centred habit-formation intervention to improve ART adherence among MSM living with HIV. We randomized 74 participants to the intervention group and 77 to the control group. We measured adherence at one, four, and ten months through medication possession ratio and self-reported adherence. Additionally, we measured viral load, CD4 cell count, major depression disorder symptoms, and alcohol and substance use disorder at baseline, fourth and tenth months. We found no statistically significant effect on adherence between groups. However, the intervention demonstrated positive results in major depression disorder symptoms (21% vs. 6%, p = 0.008) and substance use disorder (11% vs. 1%, p = 0.018) in the fourth month. The latter is relevant because, in addition to its direct benefit, it might also improve the chances of maintaining adequate adherence in the long term. This trial was retrospectively registered at ClinicalTrials.gov (trial number NCT03410680) on 8 January 2018.Trial registration: ClinicalTrials.gov identifier: NCT03410680.

Evaluation of Transparency and Openness Guidelines in Physical Therapist Journals

OBJECTIVE

The goals of this study were to evaluate the extent that physical therapist journals support open science research practices by adhering to the Transparency and Openness Promotion (TOP) guidelines and to assess the relationship between journal scores and their respective journal impact factor (JIF).

METHODS

Scimago, mapping studies, the National Library of Medicine, and journal author guidelines were searched to identify physical therapist journals for inclusion. Journals were graded on 10 standards (29 available total points) related to transparency with data, code, research materials, study design and analysis, preregistration of studies and statistical analyses, replication, and open science badges. The relationship between journal transparency and openness scores and their JIF was determined.

RESULTS

Thirty-five journals' author guidelines were assigned transparency and openness factor scores. The median score (interquartile range) across journals was 3.00 out of 29 (3.00) points (for all journals the scores ranged from 0 to 8). The 2 standards with the highest degree of implementation were design and analysis transparency (reporting guidelines) and study preregistration. No journals reported on code transparency, materials transparency, replication, and open science badges. TOP factor scores were a significant predictor of JIF scores.

CONCLUSION

There is low implementation of the TOP standards by physical therapist journals. TOP factor scores demonstrated predictive abilities for JIF scores. Policies from journals must improve to make open science practices the standard in research. Journals are in an influential position to guide practices that can improve the rigor of publication which, ultimately, enhances the evidence-based information used by physical therapists.

IMPACT

Transparent, open, and reproducible research will move the profession forward by improving the quality of research and increasing the confidence in results for implementation in clinical care.

Open Science Standards at Journals that Inform Evidence-Based Policy

Evidence-based policy uses intervention research to inform consequential decisions about resource allocation. Research findings are often published in peer-reviewed journals. Because detrimental research practices associated with closed science are common, journal articles report more false-positives and exaggerated effect sizes than would be desirable. Journal implementation of standards that promote open science-such as the transparency and openness promotion (TOP) guidelines-could reduce detrimental research practices and improve the trustworthiness of research evidence on intervention effectiveness. We evaluated TOP implementation at 339 peer-reviewed journals that have been used to identify evidence-based interventions for policymaking and programmatic decisions. Each of ten open science standards in TOP was not implemented in most journals' policies (instructions to authors), procedures (manuscript submission systems), or practices (published articles). Journals implementing at least one standard typically encouraged, but did not require, an open science practice. We discuss why and how journals could improve implementation of open science standards to safeguard evidence-based policy.

Family, Mental Health, and Placement Outcomes of a Low-cost Preventive Intervention for Youth in Foster Care

This study assessed secondary outcomes of Connecting, a low-cost, self-directed, family-based prevention program for families with youth placed in their care by state child welfare agencies. Families caring for youth aged 11 to 15 years within Washington State were recruited and randomly assigned into either the Connecting program (n = 110) or a treatment-as-usual control condition (n = 110). The program included a 10-week sequence of self-directed family activities and DVDs with video clips. Survey data were collected from caregivers and youth at baseline, immediately post-intervention, and at 12 and 24 months post-intervention; placement data was collected from the child welfare department as well. Intention-to-treat analyses focused on 5 classes of secondary outcomes at 24 months post-intervention: caregiver-youth bonding, family climate, youth risk behavior attitudes, youth mental health, and placement stability. There were no intervention effects in the full sample. In subgroup analyses, among older youth (ages 16 - 17) but not younger youth (ages 13 - 15), the Connecting condition (vs. controls) yielded more frequent caregiver-reported bonding communication, bonding activities, warmth, and positive interactions, as well as less favorable youth attitudes towards early initiation of sexual behavior and substance use, and fewer youth self-injurious thoughts. Consistent with the social development model, the divergent outcomes between younger and older youth suggests Connecting's driving mechanisms involve social processes that undergo critical shifts between early and mid-adolescence. Overall, the Connecting program showed promise in older youth for long-term promotion of caregiver-youth bonding, healthy behaviors, and mental health, but did not demonstrate long-term efficacy in facilitating stable or permanent placement of youth in care.

Money Matters: Time for Prevention and Early Intervention to Address Family Economic Circumstances

Child poverty is associated with poorer physical and mental health, negative educational outcomes and adverse long-term social and psychological consequences, all of which impact on service demand and expenditure. Until now, however, prevention and early intervention practice has tended to focus on enhancing inter-parental relationships and parenting skills (e.g., via relationship skills education, home visiting, parenting programs, family therapy) or child language, social-emotional and life skills (e.g., early childhood education, school-based programs, youth mentoring). Programs often target low-income neighborhoods or families but rarely address poverty directly. While there is substantial evidence for the effectiveness of such interventions in improving child outcomes, null results are not uncommon and even positive effects are often small, short-term, and difficult to replicate. One avenue to enhance intervention effectiveness is to improve families' economic circumstances. There are several arguments for this refocusing. It is arguably unethical to focus on individual risk without acknowledging or seeking to address (where relevant) families' social and economic contexts, while the stigma and material constraints associated with poverty can make it harder for families to engage with psychosocial support. There is also evidence that increasing household income improves child outcomes. Although national policies to alleviate poverty are important, it is increasingly recognized that practice-based initiatives have a role to play (e.g., income maximization, devolved budgets, money management support). However, knowledge about their implementation and effectiveness is relatively thin. For instance, there is some evidence that co-located welfare rights advice in healthcare settings can improve recipients' financial circumstances and health, but it is mixed and of limited quality. Moreover, there is little rigorous research on whether and how such services affect mediators (parent-child interactions, parenting capacity) and/or child physical and psychosocial outcomes directly. We call for prevention and early intervention programs to attend more to families' economic circumstances, and for experimental studies to test their implementation, reach and effectiveness.

Two-Year Risk Behavior Outcomes from Connecting, a Prevention Program for Caregivers and Youth in Foster Care

This study experimentally tested risk behavior outcomes of Connecting, a low-cost, self-directed, family-based prevention program for families with youth placed in their care by state child welfare agencies. Families caring for youth aged 11 to 15 years from across Washington State were recruited and randomly assigned to either the self-directed program with supplemental support (n = 110) or a treatment as usual control condition (n = 110). Program materials included a workbook with family activities and DVDs with video clips. Over the 10-week program, participants received motivational support contacts to prompt program completion. Survey data were collected from youth and their caregivers at baseline, directly following intervention, then again at 12 and 24 months post-intervention. Intervention effects at 24-month follow-up were found to be moderated by age. Among 16- to 17-year-old youth at follow-up, there was an intervention benefit yielding reduced use of any substance (OR = 0.71, 95% CI [0.54, 0.93], p = 0.01) and nonviolent delinquency (OR = 0.73, 95% CI [0.57, 0.94], p = 0.02). There was no intervention effect among adolescents aged 13 to 15 years for any risk behaviors. This evidence suggests that the developmental timing of a self-directed, family-focused preventive intervention for youth and their caregivers in the foster care system may influence risk behaviors that typically emerge in late adolescence. ClinicalTrials.gov Identifier: NCT03157895.

How can we optimise learning from trials in child and adolescent mental health?

Improving child and adolescent mental health requires the careful development and rigorous testing of interventions and delivery methods. This includes universal school-based mindfulness training, evaluated in the My Resilience in Adolescence (MYRIAD) trial reported in this special edition. While discovering effective interventions through randomised controlled trials is our ultimate aim, null or negative results can and should play an important role in progressing our understanding of what works. Unfortunately, alongside publication bias there can be a tendency to ignore, spin or unfairly undermine disappointing findings. This creates research waste that can increase risk and reduce benefits for future service users. We advocate several practices to help optimise learning from all trials, whatever the results: stronger intervention design reduces the likelihood of foreseeable null or negative results; an evidence-informed conceptual map of the subject area assists with understanding how results contribute to the knowledge base; mixed methods trial designs aid explanation of outcome results; various open science practices support the dispassionate analysis of data and transparent reporting of trial findings; and preparation for null or negative results helps to temper stakeholder expectations and increase understanding of why we conduct trials in the first place. To embed these practices, research funders must be willing to pay for pilot studies and 'thicker' trials, and publishers should judge trials according to their conduct and not their outcome. MYRIAD is an exemplar of how to design, conduct and report a trial to optimise learning, with important implications for practice.

Transparent, Open, and Reproducible Prevention Science

The field of prevention science aims to understand societal problems, identify effective interventions, and translate scientific evidence into policy and practice. There is growing interest among prevention scientists in the potential for transparency, openness, and reproducibility to facilitate this mission by providing opportunities to align scientific practice with scientific ideals, accelerate scientific discovery, and broaden access to scientific knowledge. The overarching goal of this manuscript is to serve as a primer introducing and providing an overview of open science for prevention researchers. In this paper, we discuss factors motivating interest in transparency and reproducibility, research practices associated with open science, and stakeholders engaged in and impacted by open science reform efforts. In addition, we discuss how and why different types of prevention research could incorporate open science practices, as well as ways that prevention science tools and methods could be leveraged to advance the wider open science movement. To promote further discussion, we conclude with potential reservations and challenges for the field of prevention science to address as it transitions to greater transparency, openness, and reproducibility. Throughout, we identify activities that aim to strengthen the reliability and efficiency of prevention science, facilitate access to its products and outputs, and promote collaborative and inclusive participation in research activities. By embracing principles of transparency, openness, and reproducibility, prevention science can better achieve its mission to advance evidence-based solutions to promote individual and collective well-being.

Using a dark logic model to explore adverse effects in audit and feedback: a qualitative study of gaming in colonoscopy

BACKGROUND

Audit and feedback (A&F) interventions improve patient care but may result in unintended consequences. To evaluate plausible harms and maximise benefits, theorisation using logic models can be useful. We aimed to explore the adverse effects of colonoscopy A&F using a feedback intervention theory (FIT) dark logic model before the National Endoscopy Database Automated Performance Reports to Improve Quality Outcomes Trial study.

METHODS

We undertook a qualitative study exploring A&F practices in colonoscopy. Interviews were undertaken with endoscopists from six English National Health Service endoscopy centres, purposively sampled for professional background and experience. A thematic framework analysis was performed, mapping paradoxical effects and harms using FIT and the theory of planned behaviour.

RESULTS

Data saturation was achieved on the 19th participant, with participants from nursing, surgical and medical backgrounds and a median of 7 years' experience.When performance was below aspirational targets participants were falsely reassured by social comparisons. Participants described confidence as a requirement for colonoscopy. Negative feedback without a plan to improve risked reducing confidence and impeding performance (cognitive interference). Unmet targets increased anxiety and prompted participants to question messages' motives and consider gaming.Participants described inaccurate documentation of subjective measures, including patient comfort, to achieve targets perceived as important. Participants described causing harm from persevering to complete procedures despite patient discomfort and removing insignificant polyps to improve detection rates without benefiting the patient.

CONCLUSION

Our dark logic model highlighted that A&F interventions may create both desired and adverse effects. Without a priori theorisation evaluations may disregard potential harms. In colonoscopy, improved patient experience measures may reduce harm. To address cognitive interference the motivation of feedback to support improvement should always be clear, with plans targeting specific behaviours and offering face-to-face support for confidence.

TRIAL REGISTRATION NUMBER

ISRCTN11126923.

Small studies, big decisions: the role of pilot/feasibility studies in incremental science and premature scale-up of behavioral interventions

Background

Careful consideration and planning are required to establish “sufficient” evidence to ensure an investment in a larger, more well-powered behavioral intervention trial is worthwhile. In the behavioral sciences, this process typically occurs where smaller-scale studies inform larger-scale trials. Believing that one can do the same things and expect the same outcomes in a larger-scale trial that were done in a smaller-scale preliminary study (i.e., pilot/feasibility) is wishful thinking, yet common practice. Starting small makes sense, but small studies come with big decisions that can influence the usefulness of the evidence designed to inform decisions about moving forward with a larger-scale trial. The purpose of this commentary is to discuss what may constitute sufficient evidence for moving forward to a definitive trial. The discussion focuses on challenges often encountered when conducting pilot/feasibility studies, referred to as common (mis)steps, that can lead to inflated estimates of both feasibility and efficacy, and how the intentional design and execution of one or more, often small, pilot/feasibility studies can play a central role in developing an intervention that scales beyond a highly localized context.

Main body

Establishing sufficient evidence to support larger-scale, definitive trials, from smaller studies, is complicated. For any given behavioral intervention, the type and amount of evidence necessary to be deemed sufficient is inherently variable and can range anywhere from qualitative interviews of individuals representative of the target population to a small-scale randomized trial that mimics the anticipated larger-scale trial. Major challenges and common (mis)steps in the execution of pilot/feasibility studies discussed are those focused on selecting the right sample size, issues with scaling, adaptations and their influence on the preliminary feasibility and efficacy estimates observed, as well as the growing pains of progressing from small to large samples. Finally, funding and resource constraints for conducting informative pilot/feasibility study(ies) are discussed.

Conclusion

Sufficient evidence to scale will always remain in the eye of the beholder. An understanding of how to design informative small pilot/feasibility studies can assist in speeding up incremental science (where everything needs to be piloted) while slowing down premature scale-up (where any evidence is sufficient for scaling).

Engaging Families in Voluntary Prevention Services to Reduce Future Child Abuse and Neglect: a Randomized Controlled Trial

Research on secondary prevention is mixed as to whether referrals to evidence-based practices and other community interventions reduce maltreatment recidivism. This evaluation sought to rigorously assess whether a community service-linkage model to evidence-based programs decreased subsequent reports of abuse and neglect among families reported to the child protection system (CPS). Between 2016 and 2018, administrative CPS records were used to identify families with at least one child under 5 years of age recently reported for maltreatment in Orange County, California, but where no follow-up services were mandated. Families were randomized to one of two community outreach and engagement conditions (i.e., targeted provider outreach vs. treatment-as-usual). Re-reporting outcomes were tracked in administrative records for a period of 12 months. A total of 4873 families met eligibility criteria, with 2231 (45.0%) randomized to the intervention arm and 2642 (54.0%) families to the control group. Among families in the intervention group, 811 (36.4%) were successfully contacted by a community provider. Of those families contacted, 145 (18.0%) agreed to participate in voluntary services. Overall, we did not detect any significant difference in re-reporting between our treatment and control groups (P = .433). Likewise, no significant differences in re-reporting were observed when we stratified the treatment group by levels of outreach and service engagement. Findings indicate low levels of successful engagement of families in voluntary services. Notwithstanding the absence of significant findings, this study demonstrates the potential for using community-based RCTs and administrative records to rigorously evaluate secondary prevention programs in the child protection system.