Reporting quality of social and psychological intervention trials: a systematic review of reporting guidelines and trial publications

A Bibliometric Analysis on Adherence to Reporting Standards for Endovascular Treatment of Chronic Lower Extremity Peripheral Artery Disease

BACKGROUND

The treatment of chronic limb-threatening ischemia (CLTI) involves a broad spectrum of therapies including many new and emerging techniques. To standardize the results of studies examining this pathology and to allow critical analysis and comparison between studies, the Society for Vascular Surgery (SVS) recommended reporting standard guidelines for the endovascular management of CLTI in 2016. Research studies that do not adhere to complete reporting standards are often more ambiguous in impact and external validity, leading to bias and misinformation that has potentially damaging effects on clinical decision-making. We thus sought to examine adherence to and factors associated with noncompliance with these recommended guidelines.

METHODS

A literature database search was conducted to include all clinical trials, randomized controlled trials, and retrospective comparative studies written in English examining the endovascular treatment of peripheral artery disease (PAD)/CLTI from January 2020 to August 2022. Systematic reviews, case reports, and meta-analysis were excluded. The manuscripts were reviewed for adherence with the SVS guidelines (overall and by guideline subcategories based on demographics, treatment methods, and outcomes), and factors associated with this adherence were determined. These data were used to calculate descriptive and comparative statistics.

RESULTS

Fifty-four manuscripts were identified from this time frame. On average, articles reviewed reported on 42.0% of the SVS reporting standards (range, 25.0-65.2%, Fig 1) with 74.1% of articles (n = 40) not adhering to at least 50.0% of the standards. Manuscripts most completely followed guidelines regarding "patient factors" and were least likely to demonstrate adherence to the description of CLTI and study complications. Within the guideline subcategories, complete adherence to guidelines was not demonstrated in any manuscript in stent trials, disease outcome measures, technical outcome measures, patient factors and critical limb ischemia description, and complete adherence rates within the other subcategories was low (range, 5.6-18.6%). Studies conducted within the United States and those with industry sponsorship were more likely to adhere to >50% of the reporting standards (P < 0.05). Journal impact factor, year of publication, and number of authors had no correlation to the percent adherence to guidelines in specific categories or adherence overall.

CONCLUSIONS

Adherence to reporting standard guidelines for endovascular treatment of lower extremity PAD specifically outlined by the SVS is suboptimal regardless of the quality of the journal the research is published in. Increasing adherence to reporting standards to provide a framework for comparison of studies across techniques used should be prioritized by authors, journal editors, and vascular societies.

Promising behavior change techniques for climate-friendly behavior change - a systematic review

Introduction

Besides societal and governmental actions to mitigate greenhouse gases, individual behavioral changes are also urgently needed to limit global temperature rise. However, these individual changes have proven to be difficult to achieve in the general population.

Methods

We conducted a systematic review in five electronic databases with the aim of systematically depicting the content of interventions that promote climate-friendly behavior in individuals and households in high- and upper-middle-income countries.

Results

We included 25 studies. The analyses included identification of the used Behavior Change Techniques (BCTs) and comparison of their promise ratio. Across our three outcome categories energy consumption, water consumption, and mobility the most frequently used BCT categories are not the ones that are most promising in terms of behavior change.

Discussion

Based on these results, our recommendation for climate change mitigation interventions is to include components that provide concrete instructions on how to perform the desired behavior (shaping knowledge), setting goals and commitments (goals and planning), substituting undesired behavior, and practicing desired behavior (repetition and substitution). Other reviews with similar aims use different wordings, definitions, or degrees of detail in their intervention component labelling which makes it difficult to compare the results. We recommend to use a standardized classification system, like the BCT taxonomy in combination with the promise ratio, which this study has shown to be a suitable tool to classify applied intervention techniques and give an indication of successful techniques.

Diversity, equity, and inclusivity in observational ambulatory assessment: Recommendations from two decades of Electronically Activated Recorder (EAR) research

Ambient audio sampling methods such as the Electronically Activated Recorder (EAR) have become increasingly prominent in clinical and social sciences research. These methods record snippets of naturalistically assessed audio from participants' daily lives, enabling novel observational research about the daily social interactions, identities, environments, behaviors, and speech of populations of interest. In practice, these scientific opportunities are equaled by methodological challenges: researchers' own cultural backgrounds and identities can easily and unknowingly permeate the collection, coding, analysis, and interpretation of social data from daily life. Ambient audio sampling poses unique and significant challenges to cultural humility, diversity, equity, and inclusivity (DEI) in scientific research that require systematized attention. Motivated by this observation, an international consortium of 21 researchers who have used ambient audio sampling methodologies created a workgroup with the aim of improving upon existing published guidelines. We pooled formally and informally documented challenges pertaining to DEI in ambient audio sampling from our collective experience on 40+ studies (most of which used the EAR app) in clinical and healthy populations ranging from children to older adults. This article presents our resultant recommendations and argues for the incorporation of community-engaged research methods in observational ambulatory assessment designs looking forward. We provide concrete recommendations across each stage typical of an ambient audio sampling study (recruiting and enrolling participants, developing coding systems, training coders, handling multi-linguistic participants, data analysis and interpretation, and dissemination of results) as well as guiding questions that can be used to adapt these recommendations to project-specific constraints and needs.

Eleven strategies for making reproducible research and open science training the norm at research institutions

Reproducible research and open science practices have the potential to accelerate scientific progress by allowing others to reuse research outputs, and by promoting rigorous research that is more likely to yield trustworthy results. However, these practices are uncommon in many fields, so there is a clear need for training that helps and encourages researchers to integrate reproducible research and open science practices into their daily work. Here, we outline eleven strategies for making training in these practices the norm at research institutions. The strategies, which emerged from a virtual brainstorming event organized in collaboration with the German Reproducibility Network, are concentrated in three areas: (i) adapting research assessment criteria and program requirements; (ii) training; (iii) building communities. We provide a brief overview of each strategy, offer tips for implementation, and provide links to resources. We also highlight the importance of allocating resources and monitoring impact. Our goal is to encourage researchers - in their roles as scientists, supervisors, mentors, instructors, and members of curriculum, hiring or evaluation committees - to think creatively about the many ways they can promote reproducible research and open science practices in their institutions.

A Systematic Review to Inform the Development of a Reporting Guideline for Concept Mapping Research

Concept mapping is a phased, mixed-method approach that is increasingly used in health research to develop an understanding of complex phenomena. The six phases of concept mapping are preparation, idea generation, structuring (clustering and prioritization), data analysis, interpretation, and utilization of the map. The reporting of concept mapping research requires the development of a specific reporting guideline. We conducted a systematic review to identify candidate reporting items for inclusion in a reporting guideline. Three databases (MEDLINE, CINAHL, and PsycInfo) were searched to identify studies that used concept mapping methodology. We included 75 concept mapping studies published since 2019 from which we extracted information about the quality of reporting. A third of the studies focused on public health. We identified 71 candidate items that relate to the quality of reporting concept mapping research. The rationale for the study, the focus prompt, procedures for brainstorming, and structuring statements were consistently reported across the included studies. The process for developing the focus prompt, the rationale for the size of the stakeholder groups, and the process for determining the final concept map were generally not reported. The findings from the review will be used to inform the development of our reporting guideline for concept mapping research.

Open Science Standards at Journals that Inform Evidence-Based Policy

Evidence-based policy uses intervention research to inform consequential decisions about resource allocation. Research findings are often published in peer-reviewed journals. Because detrimental research practices associated with closed science are common, journal articles report more false-positives and exaggerated effect sizes than would be desirable. Journal implementation of standards that promote open science-such as the transparency and openness promotion (TOP) guidelines-could reduce detrimental research practices and improve the trustworthiness of research evidence on intervention effectiveness. We evaluated TOP implementation at 339 peer-reviewed journals that have been used to identify evidence-based interventions for policymaking and programmatic decisions. Each of ten open science standards in TOP was not implemented in most journals' policies (instructions to authors), procedures (manuscript submission systems), or practices (published articles). Journals implementing at least one standard typically encouraged, but did not require, an open science practice. We discuss why and how journals could improve implementation of open science standards to safeguard evidence-based policy.

Perspectives on the application of CONSORT guidelines to randomised controlled trials in nutrition

PURPOSE

Reporting guidelines facilitate quality and completeness in research reporting. The CONsolidated Standards Of Reporting Trials (CONSORT) statement is widely applied to dietary and nutrition trials but has no extension specific to nutrition. Evidence suggests poor reporting in nutrition research. The Federation of European Nutrition Societies led an initiative to make recommendations for a nutrition extension to the CONSORT statement towards a more robust reporting of the evidence base.

METHODS

An international working group was formed of nutrition researchers from 14 institutions in 12 different countries and on five continents. Using meetings over a period of one year, we interrogated the CONSORT statement specifically for its application to report nutrition trials.

RESULTS

We provide a total of 28 new nutrition-specific recommendations or emphasised recommendations for the reporting of the introduction (three), methods (twelve), results (five) and discussion (eight). We also added two additional recommendations that were not allocated under the standard CONSORT headings.

CONCLUSION

We identify a need to provide guidance in addition to CONSORT to improve the quality and consistency of the reporting and propose key considerations for further development of formal guidelines for the reporting of nutrition trials. Readers are encouraged to engage in this process, provide comments and conduct specific studies to inform further work on the development of reporting guidelines for nutrition trials.

The influence of evidence-based program registry websites for dissemination of evidence-based interventions in behavioral healthcare

PURPOSE

Evidence-based program registries (EBPRs) are web-based databases of evaluation studies that summarize the available evidence for the effectiveness of behavioral healthcare programs, including programs addressing substance misuse, mental health, child welfare, or offender rehabilitation. The study determined the extent to which visitors to selected EBPRs accomplished the objectives of their visits and how often those visits resulted in the adoption of new or improved evidence-based interventions (EBIs).

METHOD

A follow-up telephone survey was conducted with 216 visitors to a convenience sample of six EBPRs an average of six months after the visitors' incident visit to the EBPR.

RESULTS

The most frequent objective was to identify evidence-based programs/services, curricula or assessments, followed by finding resources to implement or improve the preceding and writing a grant proposal including to comply with funding requirements; 71% of such objectives were achieved across the full set of objectives. Implementation of an EBI was completely achieved for 31% of relevant objectives and some progress on EBI implementation occurred for 19% of relevant objectives.

CONCLUSIONS

This is the first study to document the usage of EBPRs as a modality to increase the utilization of EBIs in the actual practice of behavioral healthcare. The results support the continued use of web-based EBPRs for disseminating information on evidence-based interventions for behavioral healthcare.

Survey of quality and clarity of methods and results reporting in 1 year of intervention studies published in high-impact medical and psychiatric journals

OBJECTIVE

We assessed how well articles in major medical and psychiatric journals followed best reporting practices in presenting results of intervention studies.

METHOD

Standardised data collection was used to review studies in high-impact and widely read medical (JAMA, Lancet and New England Journal of Medicine) and psychiatric (American Journal of Psychiatry, JAMA Psychiatry, Journal of Clinical Psychiatry and Lancet Psychiatry) journals, published between 1 September 2018 and 31 August 2019. Two team members independently reviewed each article.

MEASURES

The primary outcome measure was proportion of papers reporting consensus elements required to understand and evaluate the results of the intervention. The secondary outcome measure was comparison of complete and accessible reporting in the major medical versus the major psychiatric journals.

RESULTS

One hundred twenty-seven articles were identified for inclusion. At least 90% of articles in both medical and psychiatric journals included sample size, statistical significance, randomisation method, elements of study flow, and age, sex, and illness severity by randomisation group. Selected elements less frequently reported by either journal type were confidence intervals in the abstract, reported in 93% (95% CI 84% to 97%) of medical journal articles and 58% (95% CI 45% to 69%) of psychiatric journal articles, and sample size method (93%, 95% CI 84% to 97% medical; 69%, 95% CI 57% to 80% psychiatric), race and ethnicity by randomisation group (51%, 95% CI 40% to 63% medical; 73%, 95% CI 60% to 83% psychiatric), and adverse events (94%; 95% CI 86% to 98% medical; 80%, 95% CI 68% to 88% psychiatric) in the main text. CIs were included less often in psychiatric than medical journals (p<0.004 abstract, p=0.04 main text, after multiple-testing correction).

CONCLUSIONS

Recommendations include standard inclusion of a table specifying the outcome(s) designated as primary, and the sample size, effect size(s), CI(s) and p value(s) corresponding to the primary test(s) for efficacy.

Teachers’ Beliefs About Inclusive Education and Insights on What Contributes to Those Beliefs: a Meta-analytical Study

Teachers’ belief systems about the inclusion of students with special needs may explain gaps between policy and practice. We investigated three inter-related aspects of teachers’ belief systems: teachers’ cognitive appraisals (e.g., attitudes), emotional appraisal (e.g., feelings), and self-efficacy (e.g., agency to teach inclusive classrooms). To date, research in this field has produced contradictory findings, resulting in a sparse understanding of why teachers differ in their belief systems about inclusive education, and how teachers’ training experiences contribute to their development of professional beliefs. We used meta-analysis to describe the level and range of teachers’ beliefs about inclusive education, and examine factors that contribute to variation in teachers’ beliefs, namely (1) the point in teachers’ career (pre-service versus in-service), (2) training in special versus regular education, and (3) the effects of specific programs and interventions. We reviewed 102 papers (2000–2020) resulting in 191 effect sizes based on research with 40,898 teachers in 40 countries. On average, teachers’ cognitive appraisals, emotional appraisals, and efficacy about inclusion were found to be in the mid-range of scales, indicating room for growth. Self-efficacy beliefs were higher for preservice (M = 3.69) than for in-service teachers (M = 3.13). Teachers with special education training held more positive views about inclusion than regular education teachers (d = 0.41). Training and interventions related to improved cognitive appraisal (d = 0.63), emotional appraisal (d = 0.63), and self-efficacy toward inclusive practices (d = 0.93). The training was particularly effective in encouraging reflection of beliefs and, eventually, facilitating belief change when teachers gained practical experience in inclusive classrooms. Six key findings direct the next steps.

Reporting quality for abstracts of randomised trials on child and adolescent depression prevention: a meta-epidemiological study on adherence to CONSORT for abstracts

OBJECTIVES

This study aimed to investigate adherence to Consolidated Standards of Reporting Trials (CONSORT) for abstracts in reports of randomised trials on child and adolescent depression prevention. Secondary objective was to examine factors associated with overall reporting quality.

DESIGN

Meta-epidemiological study.

DATA SOURCES

We searched MEDLINE, EMBASE, PsycINFO, PsycArticles and CENTRAL.

ELIGIBILITY CRITERIA

Trials were eligible if the sample consisted of children and adolescents under 18 years with or without an increased risk for depression or subthreshold depression. We included reports published from 1 January 2003 to 8 August 2020 on randomised controlled trials (RCTs) and cluster randomised trials (CRTs) assessing universal, selective and indicated interventions aiming to prevent the onset of depression or reducing depressive symptoms.

DATA EXTRACTION AND SYNTHESIS

As the primary outcome measure, we assessed for each trial abstract whether information recommended by CONSORT was adequately reported, inadequately reported or not reported. Moreover, we calculated a summative score of overall reporting quality and analysed associations with trial and journal characteristics.

RESULTS

We identified 169 eligible studies, 103 (61%) RCTs and 66 (39%) CRTs. Adequate reporting varied considerably across CONSORT items: while 9 out of 10 abstracts adequately reported the study objective, no abstract adequately provided information on blinding. Important adverse events or side effects were only adequately reported in one out of 169 abstracts. Summative scores for the abstracts' overall reporting quality ranged from 17% to 83%, with a median of 40%. Scores were associated with the number of authors, abstract word count, journal impact factor, year of publication and abstract structure.

CONCLUSIONS

Reporting quality for abstracts of trials on child and adolescent depression prevention is suboptimal. To help health professionals make informed judgements, efforts for improving adherence to reporting guidelines for abstracts are needed.

Clearinghouse Standards of Evidence on the Transparency, Openness, and Reproducibility of Intervention Evaluations

Clearinghouses are influential repositories of information on the effectiveness of social interventions. To identify which interventions are "evidence-based," clearinghouses review intervention evaluations using published standards of evidence that focus primarily on internal validity and causal inferences. Open science practices can improve trust in evidence from evaluations on the effectiveness of social interventions. Including open science practices in clearinghouse standards of evidence is one of many efforts that could increase confidence in designations of interventions as "evidence-based." In this study, we examined the policies, procedures, and practices of 10 federal evidence clearinghouses that review preventive interventions-an important and influential subset of all evidence clearinghouses. We found that seven consider at least one open science practice when evaluating interventions: replication (6 of 10 clearinghouses), public availability of results (6), investigator conflicts of interest (3), design and analysis transparency (3), study registration (2), and protocol sharing (1). We did not identify any policies, procedures, or practices related to analysis plan registration, data sharing, code sharing, material sharing, and citation standards. We provide a framework with specific recommendations to help federal and other evidence clearinghouses implement the Transparency and Openness Promotion (TOP) Guidelines. Our proposed "TOP Guidelines for Clearinghouses" includes reporting whether evaluations used open science practices, incorporating open science practices in their standards for receiving "evidence-based" designations, and verifying that evaluations used open science practices. Doing so could increase the trustworthiness of evidence used for policy making and support improvements throughout the evidence ecosystem.

Strengthening causal inference from randomised controlled trials of complex interventions

Researchers conducting randomised controlled trials (RCTs) of complex interventions face design and analytical challenges that are not fully addressed in existing guidelines. Further guidance is needed to help ensure that these trials of complex interventions are conducted to the highest scientific standards while maximising the evidence that can be extracted from each trial. The key challenge is how to manage the multiplicity of outcomes required for the trial while minimising false positive and false negative findings. To address this challenge, we formulate three principles to conduct RCTs: (1) outcomes chosen should be driven by the intent and programme theory of the intervention and should thus be linked to testable hypotheses; (2) outcomes should be adequately powered and (3) researchers must be explicit and fully transparent about all outcomes and hypotheses before the trial is started and when the results are reported. Multiplicity in trials of complex interventions should be managed through careful planning and interpretation rather than through post hoc analytical adjustment. For trials of complex interventions, the distinction between primary and secondary outcomes as defined in current guidelines does not adequately protect against false positive and negative findings. Primary outcomes should be defined as outcomes that are relevant based on the intervention intent and programme theory, declared (ie, registered), and adequately powered. The possibility of confirmatory causal inference is limited to these outcomes. All other outcomes (either undeclared and/or inadequately powered) are secondary and inference relative to these outcomes will be exploratory.

Personalised multicomponent interventions for tobacco dependence management in low socioeconomic populations: a systematic review and meta-analysis

BACKGROUND

There remains a disproportionally high tobacco smoking rate in low-income populations. Multicomponent tobacco dependence interventions in theory are effective. However, which intervention components are necessary to include for low socioeconomic status (SES) populations is still unknown.

OBJECTIVE

To assess the effectiveness of multicomponent tobacco dependence interventions for low SES and create a checklist tool examining multicomponent interventions.

METHODS

EMBASE and MEDLINE databases were searched to identify randomised controlled trials (RCTs) published with the primary outcome of tobacco smoking cessation measured at 6 months or post intervention. RCTs that evaluated tobacco dependence management interventions (for reduction or cessation) in low SES (experience of housing insecurity, poverty, low income, unemployment, mental health challenges, illicit substance use and/or food insecurity) were included. Two authors independently abstracted data. Random effects meta-analysis and post hoc sensitivity analysis were performed.

RESULTS

Of the 33 included studies, the number of intervention components ranged from 1 to 6, with smoking quit rates varying between 1% and 36.6%. Meta-analysis revealed that both the 6-month and 12-month outcome timepoints, multicomponent interventions were successful in achieving higher smoking quit rates than the control (OR 1.64, 95% Cl 1.41 to 1.91; OR 1.74, 95% Cl 1.30 to 2.33). Evidence of low heterogeneity in the effect size was observed at 6-month (I2=26%) and moderate heterogeneity at 12-month (I2=56%) outcomes.

CONCLUSION

Multicomponent tobacco dependence interventions should focus on inclusion of social support, frequency and duration of components. Employing community-based participatory-action research approach is essential to addressing underlying psychosocioeconomic-structural factors, in addition to the proven combination pharmacotherapies.

PROSPERO REGISTRATION NUMBER

CRD42017076650.

Transparent, Open, and Reproducible Prevention Science

The field of prevention science aims to understand societal problems, identify effective interventions, and translate scientific evidence into policy and practice. There is growing interest among prevention scientists in the potential for transparency, openness, and reproducibility to facilitate this mission by providing opportunities to align scientific practice with scientific ideals, accelerate scientific discovery, and broaden access to scientific knowledge. The overarching goal of this manuscript is to serve as a primer introducing and providing an overview of open science for prevention researchers. In this paper, we discuss factors motivating interest in transparency and reproducibility, research practices associated with open science, and stakeholders engaged in and impacted by open science reform efforts. In addition, we discuss how and why different types of prevention research could incorporate open science practices, as well as ways that prevention science tools and methods could be leveraged to advance the wider open science movement. To promote further discussion, we conclude with potential reservations and challenges for the field of prevention science to address as it transitions to greater transparency, openness, and reproducibility. Throughout, we identify activities that aim to strengthen the reliability and efficiency of prevention science, facilitate access to its products and outputs, and promote collaborative and inclusive participation in research activities. By embracing principles of transparency, openness, and reproducibility, prevention science can better achieve its mission to advance evidence-based solutions to promote individual and collective well-being.

Systematic review and meta-analyses of studies analysing instructions to authors from 1987 to 2017

To gain insight into changes of scholarly journals' recommendations, we conducted a systematic review of studies that analysed journals' Instructions to Authors (ItAs). We summarised results of 153 studies, and meta-analysed how often ItAs addressed: 1) authorship, 2) conflicts of interest, 3) data sharing, 4) ethics approval, 5) funding disclosure, and 6) International Committee of Medical Journal Editors' Uniform Requirements for Manuscripts. For each topic we found large between-study heterogeneity. Here, we show six factors that explained most of that heterogeneity: 1) time (addressing of topics generally increased over time), 2) country (large differences found between countries), 3) database indexation (large differences found between databases), 4) impact factor (topics were more often addressed in highest than in lowest impact factor journals), 5) discipline (topics were more often addressed in Health Sciences than in other disciplines), and 6) sub-discipline (topics were more often addressed in general than in sub-disciplinary journals).

Clearinghouse Standards of Evidence on the Transparency, Openness, and Reproducibility of Intervention Evaluations

Clearinghouses are influential repositories of information on the effectiveness of social interventions. To identify which interventions are “evidence-based,” clearinghouses review intervention evaluations using published standards of evidence that focus primarily on internal validity and causal inferences. Open science practices can improve trust in evidence from evaluations on the effectiveness of social interventions. Including open science practices in clearinghouse standards of evidence is one of many efforts that could increase confidence in designations of interventions as “evidence-based.” In this study, we examined the policies, procedures, and practices of 10 federal evidence clearinghouses that review preventive interventions—an important and influential subset of all evidence clearinghouses. We found that seven consider at least one open science practice when evaluating interventions: replication (6 of 10 clearinghouses), public availability of results (6), investigator conflicts of interest (3), design and analysis transparency (3), study registration (2), and protocol sharing (1). We did not identify any policies, procedures, or practices related to analysis plan registration, data sharing, code sharing, material sharing, and citation standards. We provide a framework with specific recommendations to help federal and other evidence clearinghouses implement the Transparency and Openness Promotion (TOP) Guidelines. Our proposed “TOP Guidelines for Clearinghouses” includes reporting whether evaluations used open science practices, incorporating open science practices in their standards for receiving “evidence-based” designations, and verifying that evaluations used open science practices. Doing so could increase the trustworthiness of evidence used for policy making and support improvements throughout the evidence ecosystem.

How Instructional Design Is Operationalized in Various Industries for job-Seeking Learning Designers: Engaging the Talent Development Capability Model

In 2020, as COVID-19 impacted the world, instructional designers quickly came to the forefront of higher education, consulting, and corporate settings. Additionally, in early 2020, the Association for Talent Development (ATD) globally launched their new Talent Development Capability Model. To determine how instructional design is operationalized in various industries, we utilized the Talent Development Capability Model as a framework to review 100 online job postings from LinkedIn, Indeed.com, and HigherEdJobs.com, scanning each for the 23 capabilities identified in the ATD Talent Development Capability Model. Using the preferred reporting items for systematic reviews and meta-analyses (PRISMA) organizational framework as our method of reviewing job postings, we found that instructional design, talent delivery and facilitation, technology application, communication, and collaboration and leadership capabilities appeared the most frequently. Educational requirements were statistically significantly associated with job sectors, as shown in our analysis. With these results and analysis, this research will inform both formal instructional design education programs and job seekers regarding how instructional design is operationalized in job requisitions. This research will also provide insights into alignments with the ATD Talent Development Capability Model and job requisitions.

Common Methodological Problems in Randomized Controlled Trials of Preventive Interventions

Randomized controlled trials (RCTs) are often considered the gold standard in evaluating whether intervention results are in line with causal claims of beneficial effects. However, given that poor design and incorrect analysis may lead to biased outcomes, simply employing an RCT is not enough to say an intervention "works." This paper applies a subset of the Society for Prevention Research (SPR) Standards of Evidence for Efficacy, Effectiveness, and Scale-up Research, with a focus on internal validity (making causal inferences) to determine the degree to which RCTs of preventive interventions are well-designed and analyzed, and whether authors provide a clear description of the methods used to report their study findings. We conducted a descriptive analysis of 851 RCTs published from 2010 to 2020 and reviewed by the Blueprints for Healthy Youth Development web-based registry of scientifically proven and scalable interventions. We used Blueprints' evaluation criteria that correspond to a subset of SPR's standards of evidence. Only 22% of the sample satisfied important criteria for minimizing biases that threaten internal validity. Overall, we identified an average of 1-2 methodological weaknesses per RCT. The most frequent sources of bias were problems related to baseline non-equivalence (i.e., differences between conditions at randomization) or differential attrition (i.e., differences between completers versus attritors or differences between study conditions that may compromise the randomization). Additionally, over half the sample (51%) had missing or incomplete tests to rule out these potential sources of bias. Most preventive intervention RCTs need improvement in rigor to permit causal inference claims that an intervention is effective. Researchers also must improve reporting of methods and results to fully assess methodological quality. These advancements will increase the usefulness of preventive interventions by ensuring the credibility and usability of RCT findings.

Evaluating implementation of the Transparency and Openness Promotion (TOP) guidelines: the TRUST process for rating journal policies, procedures, and practices

BACKGROUND

The Transparency and Openness Promotion (TOP) Guidelines describe modular standards that journals can adopt to promote open science. The TOP Factor is a metric to describe the extent to which journals have adopted the TOP Guidelines in their policies. Systematic methods and rating instruments are needed to calculate the TOP Factor. Moreover, implementation of these open science policies depends on journal procedures and practices, for which TOP provides no standards or rating instruments.

METHODS

We describe a process for assessing journal policies, procedures, and practices according to the TOP Guidelines. We developed this process as part of the Transparency of Research Underpinning Social Intervention Tiers (TRUST) Initiative to advance open science in the social intervention research ecosystem. We also provide new instruments for rating journal instructions to authors (policies), manuscript submission systems (procedures), and published articles (practices) according to standards in the TOP Guidelines. In addition, we describe how to determine the TOP Factor score for a journal, calculate reliability of journal ratings, and assess coherence among a journal's policies, procedures, and practices. As a demonstration of this process, we describe a protocol for studying approximately 345 influential journals that have published research used to inform evidence-based policy.

DISCUSSION

The TRUST Process includes systematic methods and rating instruments for assessing and facilitating implementation of the TOP Guidelines by journals across disciplines. Our study of journals publishing influential social intervention research will provide a comprehensive account of whether these journals have policies, procedures, and practices that are consistent with standards for open science and thereby facilitate the publication of trustworthy findings to inform evidence-based policy. Through this demonstration, we expect to identify ways to refine the TOP Guidelines and the TOP Factor. Refinements could include: improving templates for adoption in journal instructions to authors, manuscript submission systems, and published articles; revising explanatory guidance intended to enhance the use, understanding, and dissemination of the TOP Guidelines; and clarifying the distinctions among different levels of implementation. Research materials are available on the Open Science Framework: https://osf.io/txyr3/ .

A systematic review and meta-analysis of psychological interventions to improve mental wellbeing

Our current understanding of the efficacy of psychological interventions in improving mental states of wellbeing is incomplete. This study aimed to overcome limitations of previous reviews by examining the efficacy of distinct types of psychological interventions, irrespective of their theoretical underpinning, and the impact of various moderators, in a unified systematic review and meta-analysis. Four-hundred-and-nineteen randomized controlled trials from clinical and non-clinical populations (n = 53,288) were identified for inclusion. Mindfulness-based and multi-component positive psychological interventions demonstrated the greatest efficacy in both clinical and non-clinical populations. Meta-analyses also found that singular positive psychological interventions, cognitive and behavioural therapy-based, acceptance and commitment therapy-based, and reminiscence interventions were impactful. Effect sizes were moderate at best, but differed according to target population and moderator, most notably intervention intensity. The evidence quality was generally low to moderate. While the evidence requires further advancement, the review provides insight into how psychological interventions can be designed to improve mental wellbeing.

Drawing on Dialogues in Arts-Based Dynamic Interpersonal Therapy (ADIT) for Complex Depression: A Complex Intervention Development Study Using the Medical Research Council (UK) Phased Guidance

Aim: The aim of this paper is to present the development and evaluation of an art psychotherapy brief treatment method for complex depression for patients referred to mental health services.

Background: Art Psychotherapy literature describes a range of processes of relational change through the use of arts focused and relationship focused interventions. Complex depression has a prevalence of 3% of the population in the West and it is recorded that in 2016 only 28% of that population were receiving psychological treatment. This study was developed to test the hypothesis of whether an accessible and acceptable approach to the treatment of complex depression could be developed in relation to existing evidence-based practice within mental health services.

Method: The United Kingdom Medical Research Council phased guidance for complex intervention development was used (Phases I and II) to develop the intervention. The process included producing a literature overview, systematic description of clinical practice, including a logic model and a clinical protocol. The art psychotherapy protocol described an arts-based dynamic interpersonal therapy approach (ADIT), offered 1:1 over 24 sessions. Further to this the intervention was tested for referrer acceptability. The intervention is in the early stages of evaluation, using changes to the patient's depression and anxiety measured pre- and post-treatment with a follow-up measure at 3 months following completion of treatment.

Results: Phase I of the study provided a good basis for developing a logic model and protocol. The authors found that there was good clinical consensus about the use of a structured clinical art psychotherapy method (ADIT) and the literature overview was used to support specific examples of good practice. The verification of clinical coherence was represented by a logic model and clinical protocol for delivering the intervention. The acceptability study demonstrated very high levels of acceptability for referrers reporting that (i) ADIT was acceptable for patients with complex/major depression, (ii) that they were likely to refer to ADIT in the future (iii) that the use of arts was likely to improve accessibility (iv) the use of arts was likely to improve outcomes and (v) that offering ADIT was an effective use of mental health resources.

Discussion: Phase I of this intervention development study (following MRC guidance) demonstrated theoretical and practice coherence resulting in a clinical protocol and logic model. Whilst Phase II of this study showed promising results, Phase II would need to be sufficiently scaled up to a full trial to further test the intervention and protocol.

Improving open and rigorous science: ten key future research opportunities related to rigor, reproducibility, and transparency in scientific research

Background: As part of a coordinated effort to expand research activity around rigor, reproducibility, and transparency (RRT) across scientific disciplines, a team of investigators at the Indiana University School of Public Health-Bloomington hosted a workshop in October 2019 with international leaders to discuss key opportunities for RRT research. Objective: The workshop aimed to identify research priorities and opportunities related to RRT. Design: Over two-days, workshop attendees gave presentations and participated in three working groups: (1) Improving Education & Training in RRT, (2) Reducing Statistical Errors and Increasing Analytic Transparency, and (3) Looking Outward: Increasing Truthfulness and Accuracy of Research Communications. Following small-group discussions, the working groups presented their findings, and participants discussed the research opportunities identified. The investigators compiled a list of research priorities, which were circulated to all participants for feedback. Results: Participants identified the following priority research questions: (1) Can RRT-focused statistics and mathematical modeling courses improve statistics practice?; (2) Can specialized training in scientific writing improve transparency?; (3) Does modality (e.g. face to face, online) affect the efficacy RRT-related education?; (4) How can automated programs help identify errors more efficiently?; (5) What is the prevalence and impact of errors in scientific publications (e.g., analytic inconsistencies, statistical errors, and other objective errors)?; (6) Do error prevention workflows reduce errors?; (7) How do we encourage post-publication error correction?; (8) How does 'spin' in research communication affect stakeholder understanding and use of research evidence?; (9) Do tools to aid writing research reports increase comprehensiveness and clarity of research reports?; and (10) Is it possible to inculcate scientific values and norms related to truthful, rigorous, accurate, and comprehensive scientific reporting? Conclusion: Participants identified important and relatively unexplored questions related to improving RRT. This list may be useful to the scientific community and investigators seeking to advance meta-science (i.e. research on research).

mHealth app using machine learning to increase physical activity in diabetes and depression: clinical trial protocol for the DIAMANTE Study

INTRODUCTION

Depression and diabetes are highly disabling diseases with a high prevalence and high rate of comorbidity, particularly in low-income ethnic minority patients. Though comorbidity increases the risk of adverse outcomes and mortality, most clinical interventions target these diseases separately. Increasing physical activity might be effective to simultaneously lower depressive symptoms and improve glycaemic control. Self-management apps are a cost-effective, scalable and easy access treatment to increase physical activity. However, cutting-edge technological applications often do not reach vulnerable populations and are not tailored to an individual's behaviour and characteristics. Tailoring of interventions using machine learning methods likely increases the effectiveness of the intervention.

METHODS AND ANALYSIS

In a three-arm randomised controlled trial, we will examine the effect of a text-messaging smartphone application to encourage physical activity in low-income ethnic minority patients with comorbid diabetes and depression. The adaptive intervention group receives messages chosen from different messaging banks by a reinforcement learning algorithm. The uniform random intervention group receives the same messages, but chosen from the messaging banks with equal probabilities. The control group receives a weekly mood message. We aim to recruit 276 adults from primary care clinics aged 18-75 years who have been diagnosed with current diabetes and show elevated depressive symptoms (Patient Health Questionnaire depression scale-8 (PHQ-8) >5). We will compare passively collected daily step counts, self-report PHQ-8 and most recent haemoglobin A1c from medical records at baseline and at intervention completion at 6-month follow-up.

ETHICS AND DISSEMINATION

The Institutional Review Board at the University of California San Francisco approved this study (IRB: 17-22608). We plan to submit manuscripts describing our user-designed methods and testing of the adaptive learning algorithm and will submit the results of the trial for publication in peer-reviewed journals and presentations at (inter)-national scientific meetings.

TRIAL REGISTRATION NUMBER

NCT03490253; pre-results.

Process evaluation of complex cardiovascular interventions: How to interpret the results of my trial?

Complex interventions of varying degrees of complexity are commonly used and evaluated in cardiovascular nursing and allied professions. Such interventions are increasingly tested using randomized trial designs. However, process evaluations are seldom used to better understand the results of these trials. Process evaluation aims to understand how complex interventions create change by evaluating implementation, mechanisms of impact, and the surrounding context when delivering an intervention. As such, this method can illuminate important mechanisms and clarify variation in results. In this article, process evaluation is described according to the Medical Research Council guidance and its use exemplified through a randomized controlled trial evaluating the effectiveness of a transition program for adolescents with chronic conditions.

Increase value and reduce waste in research on psychological therapies

A seminal Lancet series focused on increasing value and reducing waste in biomedical research, providing a transferrable template to diagnose problems in research. Our goal was to document how some of these sources of waste apply to mental health and particularly psychological treatments research. We synthesize and critically evaluate empirical findings in relation to four major sources: i) defining research priorities; ii) research design, methods and analysis; iii) accessibility of research information; iv) accuracy and usability of research reports. We demonstrate that each source of waste considered is well-represented and amply documented within this field. We describe hype and insufficient consideration of what is known in defining research priorities, persistent risk of bias, particularly due to selective outcome reporting, for psychotherapy trials across mental disorders, intellectual and financial biases, direct and indirect evidence of publication bias, largely inexistent adoption of data sharing, issues of multiplicity and fragmentation of data and findings, and insufficient adoption of reporting guidelines. We expand on a few general solutions, including supporting meta-research, properly testing interventions to increase research quality, placing open science at the center of psychological treatment research and remaining vigilant particularly regarding the strains of research currently prioritized, such as experimental psychopathology.

Use of publication procedures to improve research integrity by addiction journals

BACKGROUND AND AIMS

The credibility crisis evident in many academic disciplines has led peer-reviewed journals to implement procedures to reduce use of flexible data analysis practices and selective reporting of results. This exploratory study examined the adoption of six of these procedures by addiction journals.

METHODS

Thirty-eight high-impact addiction journals were identified using the 2018 Clarivate Analytics Journal Citation Report for 2017 ranks. The online instructions for authors were reviewed for references to six publication procedures: conflict of interest disclosure, reporting guidelines, clinical trial registration, registration of other study designs, data-sharing and registered reports. The webpages of the Center for Open Science and Consolidated Standards of Reporting Trials (CONSORT) were also reviewed for data pertaining to registered reports and reporting guidelines, respectively.

RESULTS

The range of procedures adopted by the addiction journals was 0-5, with a mean of 2.66. Conflict-of-interest disclosure was required by all but one journal. Encouraging data-sharing was the next most commonly required procedure. Fewer than half the journals recommended specific reporting guidelines or required registration of clinical trials, and only four required procedures to pre-specify hypotheses and analytical methods.

CONCLUSIONS

While many addiction journals have adopted publication procedures to improve research integrity, these can be limited by their voluntary nature and monitoring difficulties. More stringent requirements that lock researchers into specific hypotheses and analyses have not been widely adopted.

Community-wide intervention and population-level physical activity: a 5-year cluster randomized trial

Background

Evidence from a limited number of short-term trials indicates the difficulty in achieving population-level improvements in physical activity (PA) through community-wide interventions (CWIs). We sought to evaluate the effectiveness of a 5-year CWI for promoting PA in middle-aged and older adults using a cluster randomized design.

Methods

We randomized 12 communities in Unnan, Japan, to either intervention (9) or control (3). Additionally, intervention communities were randomly allocated to three subgroups by different PA types promoted. Randomly sampled residents aged 40–79 years responded to the baseline survey (n = 4414; 74%) and were followed at 1, 3 and 5 years (78–83% response rate). The intervention was a 5-year CWI using social marketing to promote PA. The primary outcome was a change in recommended levels of PA.

Results

Compared with control communities, adults achieving recommended levels of PA increased in intervention communities [adjusted change difference = 4.6 percentage points (95% confidence interval: 0.4, 8.8)]. The intervention was effective for promoting all types of recommended PAs, i.e. aerobic (walking, 6.4%), flexibility (6.1%) and muscle-strengthening activities (5.7%). However, a bundled approach, which attempted to promote all forms of PAs above simultaneously, was not effective (1.3–3.4%, P ≥ 0.138). Linear dose–response relationships between the CWI awareness and changes in PA were observed (P ≤ 0.02). Pain intensity decreased in shoulder (intervention and control) and lower back (intervention only) but there was little change difference in all musculoskeletal pain outcomes between the groups.

Conclusions

The 5-year CWI using the focused social marketing strategy increased the population-level of PA.

CONSORT-SPI 2018 Explanation and Elaboration: guidance for reporting social and psychological intervention trials

BACKGROUND

The CONSORT (Consolidated Standards of Reporting Trials) Statement was developed to help biomedical researchers report randomised controlled trials (RCTs) transparently. We have developed an extension to the CONSORT 2010 Statement for social and psychological interventions (CONSORT-SPI 2018) to help behavioural and social scientists report these studies transparently.

METHODS

Following a systematic review of existing reporting guidelines, we conducted an online Delphi process to prioritise the list of potential items for the CONSORT-SPI 2018 checklist identified from the systematic review. Of 384 international participants, 321 (84%) participated in both rating rounds. We then held a consensus meeting of 31 scientists, journal editors, and research funders (March 2014) to finalise the content of the CONSORT-SPI 2018 checklist and flow diagram.

RESULTS

CONSORT-SPI 2018 extends 9 items (14 including sub-items) from the CONSORT 2010 checklist, adds a new item (with 3 sub-items) related to stakeholder involvement in trials, and modifies the CONSORT 2010 flow diagram. This Explanation and Elaboration (E&E) document is a user manual to enhance understanding of CONSORT-SPI 2018. It discusses the meaning and rationale for each checklist item and provides examples of complete and transparent reporting.

CONCLUSIONS

The CONSORT-SPI 2018 Extension, this E&E document, and the CONSORT website ( www.consort-statement.org ) are helpful resources for improving the reporting of social and psychological intervention RCTs.

Rating the quality of a body of evidence on the effectiveness of health and social interventions: A systematic review and mapping of evidence domains

INTRODUCTION

Rating the quality of a body of evidence is an increasingly common component of research syntheses on intervention effectiveness. This study sought to identify and examine existing systems for rating the quality of a body of evidence on the effectiveness of health and social interventions.

METHODS

We used a multicomponent search strategy to search for full-length reports of systems for rating the quality of a body of evidence on the effectiveness of health and social interventions published in English from 1995 onward. Two independent reviewers extracted data from each eligible system on the evidence domains included, as well as the development and dissemination processes for each system.

RESULTS

Seventeen systems met our eligibility criteria. Across systems, we identified 13 discrete evidence domains: study design, study execution, consistency, measures of precision, directness, publication bias, magnitude of effect, dose-response, plausible confounding, analogy, robustness, applicability, and coherence. We found little reporting of rigorous procedures in the development and dissemination of evidence rating systems.

CONCLUSION

We identified 17 systems for rating the quality of a body of evidence on intervention effectiveness across health and social policy. Existing systems vary greatly in the domains they include and how they operationalize domains, and most have important limitations in their development and dissemination. The construct of the quality of the body of evidence was defined in a few systems largely extending the Grading of Recommendations Assessment, Development, and Evaluation approach. Grading of Recommendations Assessment, Development, and Evaluation was found to be unique in its comprehensive guidance, rigorous development, and dissemination strategy.

Standards for design and measurement would make clinical research reproducible and usable

We find standards useful in everyday life and in science, although we do not always follow them. Adopting new standards can be expensive, so there may be a strong incentive to maintain the status quo rather than adopt new standards. The scientific community has many standards encompassing both doing clinical research and reporting it, including standards for design and measurement. Although existing research standards have improved both research and its reporting, we need to unify existing standards and to fill the gaps between steps throughout the research process. Existing gaps include implementation of standards and links between standards for study registration (to know about all studies undertaken), study protocols (to identify the preplanned study design and methods), data collection (to assess outcomes that are important and comparable across studies), dissemination of findings (to know the results of previous studies), data sharing (to make best use of existing data), and evidence synthesis (to draw appropriate conclusions from the body of evidence). The scientific community must work together to harmonize existing standards, to ensure that standards are kept up to date, to check that standards are followed, and to develop standards where they are still needed. A unified system of standards will make our work more reproducible.

Scientific Reporting: Raising the Standards

This article is based on a presentation that was made at the 2014 annual meeting of the editorial board of Health Education & Behavior. The article addresses critical issues related to standards of scientific reporting in journals, including concerns about external and internal validity and reporting bias. It reviews current reporting guidelines, effects of adopting guidelines, and offers suggestions for improving reporting. The evidence about the effects of guideline adoption and implementation is briefly reviewed. Recommendations for adoption and implementation of appropriate guidelines, including considerations for journals, are provided.

Meta-Analysis of Complex Interventions

Meta-analysis is a prominent method for estimating the effects of public health interventions, yet these interventions are often complex in ways that pose challenges to using conventional meta-analytic methods. This article discusses meta-analytic techniques that can be used in research syntheses on the effects of complex public health interventions. We first introduce the use of complexity frameworks to conceptualize public health interventions. We then present a menu of meta-analytic procedures for addressing various sources of complexity when answering questions about the effects of public health interventions in research syntheses. We conclude with a review of important practices and key resources for conducting meta-analyses on complex interventions, as well as future directions for research synthesis more generally. Overall, we argue that it is possible to conduct meaningful quantitative syntheses of research on the effects of public health interventions, though these meta-analyses may require the use of advanced techniques to properly consider and attend to issues of complexity.

Risk of bias in overviews of reviews: a scoping review of methodological guidance and four-item checklist

OBJECTIVE

To assess the conditions under which employing an overview of systematic reviews is likely to lead to a high risk of bias.

STUDY DESIGN

To synthesise existing guidance concerning overview practice, a scoping review was conducted. Four electronic databases were searched with a pre-specified strategy (PROSPERO 2015:CRD42015027592) ending October 2015. Included studies needed to describe or develop overview methodology. Data were narratively synthesised to delineate areas highlighted as outstanding challenges or where methodological recommendations conflict.

RESULTS

Twenty-four papers met the inclusion criteria. There is emerging debate regarding overlapping systematic reviews; systematic review scope; quality of included research; updating; and synthesizing and reporting results. While three functions for overviews have been proposed-identify gaps, explore heterogeneity, summarize evidence-overviews cannot perform the first; are unlikely to achieve the second and third simultaneously; and can only perform the third under specific circumstances. Namely, when identified systematic reviews meet the following four conditions: (1) include primary trials that do not substantially overlap, (2) match overview scope, (3) are of high methodological quality, and (4) are up-to-date.

CONCLUSION

Considering the intended function of proposed overviews with the corresponding methodological conditions may improve the quality of this burgeoning publication type. Copyright © 2017 John Wiley & Sons, Ltd.

Reporting quality and risk of bias in randomised trials in health professions education

CONTEXT

Complete reporting of research is essential to enable consumers to accurately appraise, interpret and apply findings. Quality appraisal checklists are giving way to tools that judge the risk for bias.

OBJECTIVES

We sought to determine the prevalence of these complementary aspects of research reports (completeness of reporting and perceived risk for bias) of randomised studies in health professions education.

METHODS

We searched bibliographic databases for randomised studies of health professions education. We appraised two cohorts representing different time periods (2008-2010 and 2014, respectively) and worked in duplicate to apply the CONSORT guidelines and Cochrane Risk of Bias tool. We explored differences between time periods using independent-samples t-tests or the chi-squared test, as appropriate.

RESULTS

We systematically identified 180 randomised studies (2008-2010, n = 150; 2014, n = 30). Frequencies of reporting of CONSORT elements within full-text reports were highly variable and most elements were reported in fewer than 50% of studies. We found a statistically significant difference in the CONSORT reporting index (maximum score: 500) between the 2008-2010 (mean ± standard deviation [SD]: 242.7 ± 55.6) and 2014 (mean ± SD: 311.6 ± 53.2) cohorts (p < 0.001). High or unclear risk for bias was most common for allocation concealment (157, 87%) and blinding of participants (147, 82%), personnel (152, 84%) and outcome assessors (112, 62%). Most risk for bias elements were judged to be unclear (range: 51-84%). Risk for bias elements significantly improved over time for blinding of participants (p = 0.007), incomplete data (p < 0.001) and the presence of other sources of bias (p < 0.001).

CONCLUSIONS

Reports of randomised studies in health professions education frequently omit elements recommended by the CONSORT statement. Most reports were assessed as having a high or unclear risk for bias. Greater attention to how studies are reported at study outset and in manuscript preparation could improve levels of complete transparent reporting.

Systematic review of interventions for Indigenous adults with mental and substance use disorders in Australia, Canada, New Zealand and the United States

OBJECTIVE

The aim of this study was to systematically review the evidence-base for the effectiveness of culturally unadapted, culturally adapted and culture-based interventions for Indigenous adults with mental or substance use disorders.

METHODS

We conducted a systematic search of scientific databases, government websites and web-based Indigenous research repositories. We sought studies using designs comparing an intervention group to a control/comparator group or pre- and post-test designs, published between 2000 and 2015 examining interventions to improve individual-level outcomes (e.g. remission, symptoms, quality of life, functioning) or service-level outcomes (e.g. number of interventions delivered) for Indigenous adults with mental or substance use disorders in Australia, Canada, New Zealand or the United States.

RESULTS

A total of 16 studies met inclusion criteria. Virtually all North American studies (6 US and 1 Canadian) evaluated culturally unadapted interventions, all of which were interventions for substance use. Two-thirds of Australian and New Zealand studies evaluated culturally adapted interventions and included samples with mental disorders. Of eight culturally unadapted psychological/psychosocial, pharmacological and educational intervention studies, seven reported significant improvements on at least one measure of psychological well-being, mental health problem severity, or significantly reduced alcohol or illicit drug use. Of seven culturally adapted psychological/psychosocial intervention studies, all reported significant improvement on at least one measure of symptoms of mental illness, functioning, and alcohol use. One culture-based psychological/psychosocial intervention study significantly reduced problem severity in medical and psychiatric domains.

CONCLUSION

There remains inconclusive evidence regarding interventions due to a small and methodologically weak evidence-base. The literature would be enhanced by intervention replication and outcome standardisation, validating the outcome instruments used in Indigenous populations, including sample size calculations and using stronger research designs (e.g. interrupted time-series designs). Robust implementation and outcomes research is needed to further progress evidence-based practice in Indigenous mental health.

Reviewing and interpreting the effects of brief alcohol interventions: comment on a Cochrane review about motivational interviewing for young adults

BACKGROUND

Cochrane recently published a systematic review on motivational interviewing (MI) for alcohol misuse in young adults. The review authors concluded that 'there are no substantive, meaningful benefits of MI interventions for the prevention of alcohol misuse' (p. 2), as effect sizes were 'small and unlikely to be of any meaningful benefit in practice' (p. 27). As most of these interventions were quite brief, we wish to open a dialogue about interpreting effect sizes in this review and of (brief) alcohol interventions more generally.

ANALYSIS

We analyze four methodological aspects of the review that likely influenced the author's conclusions about intervention effects: (1) risk of bias assessments, (2) search strategies, (3) assessing the quality of the body of evidence and (4) definitions of sustainability and clinical significance.

CONCLUSIONS

We interpret the effect sizes found in this review to indicate modest yet beneficial and potentially meaningful effects of these interventions, given their brevity and low cost. This interpretation is consistent with other reviews on brief, MI-based interventions and brief interventions more generally. We therefore encourage the field to re-open dialogue about the clinical importance of the effects of MI on alcohol misuse by young adults. Rather than dismissing interventions with small effects, we believe a more fruitful way forward for the field would be to catalogue effect sizes for various alcohol interventions. Such a catalogue would help stakeholders themselves to choose which interventions meet their minimum desired impact, and thus may be suitable given their targeted populations, setting and resources.

Increasing value and reducing waste in biomedical research: who's listening?

The biomedical research complex has been estimated to consume almost a quarter of a trillion US dollars every year. Unfortunately, evidence suggests that a high proportion of this sum is avoidably wasted. In 2014, The Lancet published a series of five reviews showing how dividends from the investment in research might be increased from the relevance and priorities of the questions being asked, to how the research is designed, conducted, and reported. 17 recommendations were addressed to five main stakeholders-funders, regulators, journals, academic institutions, and researchers. This Review provides some initial observations on the possible effects of the Series, which seems to have provoked several important discussions and is on the agendas of several key players. Some examples of individual initiatives show ways to reduce waste and increase value in biomedical research. This momentum will probably move strongly across stakeholder groups, if collaborative relationships evolve between key players; further important work is needed to increase research value. A forthcoming meeting in Edinburgh, UK, will provide an initial forum within which to foster the collaboration needed.

Standards of Evidence for Efficacy, Effectiveness, and Scale-up Research in Prevention Science: Next Generation

A decade ago, the Society of Prevention Research (SPR) endorsed a set of standards for evidence related to research on prevention interventions. These standards (Flay et al., Prevention Science 6:151-175, 2005) were intended in part to increase consistency in reviews of prevention research that often generated disparate lists of effective interventions due to the application of different standards for what was considered to be necessary to demonstrate effectiveness. In 2013, SPR's Board of Directors decided that the field has progressed sufficiently to warrant a review and, if necessary, publication of "the next generation" of standards of evidence. The Board convened a committee to review and update the standards. This article reports on the results of this committee's deliberations, summarizing changes made to the earlier standards and explaining the rationale for each change. The SPR Board of Directors endorses "The Standards of Evidence for Efficacy, Effectiveness, and Scale-up Research in Prevention Science: Next Generation."

Standards of Evidence for Efficacy, Effectiveness, and Scale-up Research in Prevention Science: Next Generation

A decade ago, the Society of Prevention Research (SPR) endorsed a set of standards for evidence related to research on prevention interventions. These standards (Flay et al., Prevention Science 6:151-175, 2005) were intended in part to increase consistency in reviews of prevention research that often generated disparate lists of effective interventions due to the application of different standards for what was considered to be necessary to demonstrate effectiveness. In 2013, SPR's Board of Directors decided that the field has progressed sufficiently to warrant a review and, if necessary, publication of "the next generation" of standards of evidence. The Board convened a committee to review and update the standards. This article reports on the results of this committee's deliberations, summarizing changes made to the earlier standards and explaining the rationale for each change. The SPR Board of Directors endorses "The Standards of Evidence for Efficacy, Effectiveness, and Scale-up Research in Prevention Science: Next Generation."

Portion, package or tableware size for changing selection and consumption of food, alcohol and tobacco

BACKGROUND

Overeating and harmful alcohol and tobacco use have been linked to the aetiology of various non-communicable diseases, which are among the leading global causes of morbidity and premature mortality. As people are repeatedly exposed to varying sizes and shapes of food, alcohol and tobacco products in environments such as shops, restaurants, bars and homes, this has stimulated public health policy interest in product size and shape as potential targets for intervention.

OBJECTIVES

1) To assess the effects of interventions involving exposure to different sizes or sets of physical dimensions of a portion, package, individual unit or item of tableware on unregulated selection or consumption of food, alcohol or tobacco products in adults and children.2) To assess the extent to which these effects may be modified by study, intervention and participant characteristics.

SEARCH METHODS

We searched CENTRAL, MEDLINE, EMBASE, PsycINFO, eight other published or grey literature databases, trial registries and key websites up to November 2012, followed by citation searches and contacts with study authors. This original search identified eligible studies published up to July 2013, which are fully incorporated into the review. We conducted an updated search up to 30 January 2015 but further eligible studies are not yet fully incorporated due to their minimal potential to change the conclusions.

SELECTION CRITERIA

Randomised controlled trials with between-subjects (parallel-group) or within-subjects (cross-over) designs, conducted in laboratory or field settings, in adults or children. Eligible studies compared at least two groups of participants, each exposed to a different size or shape of a portion of a food (including non-alcoholic beverages), alcohol or tobacco product, its package or individual unit size, or of an item of tableware used to consume it, and included a measure of unregulated selection or consumption of food, alcohol or tobacco.

DATA COLLECTION AND ANALYSIS

We applied standard Cochrane methods to select eligible studies for inclusion and to collect data and assess risk of bias. We calculated study-level effect sizes as standardised mean differences (SMDs) between comparison groups, measured as quantities selected or consumed. We combined these results using random-effects meta-analysis models to estimate summary effect sizes (SMDs with 95% confidence intervals (CIs)) for each outcome for size and shape comparisons. We rated the overall quality of evidence using the GRADE system. Finally, we used meta-regression analysis to investigate statistical associations between summary effect sizes and variant study, intervention or participant characteristics.

MAIN RESULTS

The current version of this review includes 72 studies, published between 1978 and July 2013, assessed as being at overall unclear or high risk of bias with respect to selection and consumption outcomes. Ninety-six per cent of included studies (69/72) manipulated food products and 4% (3/72) manipulated cigarettes. No included studies manipulated alcohol products. Forty-nine per cent (35/72) manipulated portion size, 14% (10/72) package size and 21% (15/72) tableware size or shape. More studies investigated effects among adults (76% (55/72)) than children and all studies were conducted in high-income countries - predominantly in the USA (81% (58/72)). Sources of funding were reported for the majority of studies, with no evidence of funding by agencies with possible commercial interests in their results.A meta-analysis of 86 independent comparisons from 58 studies (6603 participants) found a small to moderate effect of portion, package, individual unit or tableware size on consumption of food (SMD 0.38, 95% CI 0.29 to 0.46), providing moderate quality evidence that exposure to larger sizes increased quantities of food consumed among children (SMD 0.21, 95% CI 0.10 to 0.31) and adults (SMD 0.46, 95% CI 0.40 to 0.52). The size of this effect suggests that, if sustained reductions in exposure to larger-sized food portions, packages and tableware could be achieved across the whole diet, this could reduce average daily energy consumed from food by between 144 and 228 kcal (8.5% to 13.5% from a baseline of 1689 kcal) among UK children and adults. A meta-analysis of six independent comparisons from three studies (108 participants) found low quality evidence for no difference in the effect of cigarette length on consumption (SMD 0.25, 95% CI -0.14 to 0.65).One included study (50 participants) estimated a large effect on consumption of exposure to differently shaped tableware (SMD 1.17, 95% CI 0.57 to 1.78), rated as very low quality evidence that exposure to shorter, wider bottles (versus taller, narrower bottles) increased quantities of water consumed by young adult participants.A meta-analysis of 13 independent comparisons from 10 studies (1164 participants) found a small to moderate effect of portion or tableware size on selection of food (SMD 0.42, 95% CI 0.24 to 0.59), rated as moderate quality evidence that exposure to larger sizes increased the quantities of food people selected for subsequent consumption. This effect was present among adults (SMD 0.55, 95% CI 0.35 to 0.75) but not children (SMD 0.14, 95% CI -0.06 to 0.34).In addition, a meta-analysis of three independent comparisons from three studies (232 participants) found a very large effect of exposure to differently shaped tableware on selection of non-alcoholic beverages (SMD 1.47, 95% CI 0.52 to 2.43), rated as low quality evidence that exposure to shorter, wider (versus taller, narrower) glasses or bottles increased the quantities selected for subsequent consumption among adults (SMD 2.31, 95% CI 1.79 to 2.83) and children (SMD 1.03, 95% CI 0.41 to 1.65).

AUTHORS' CONCLUSIONS

This review found that people consistently consume more food and drink when offered larger-sized portions, packages or tableware than when offered smaller-sized versions. This suggests that policies and practices that successfully reduce the size, availability and appeal of larger-sized portions, packages, individual units and tableware can contribute to meaningful reductions in the quantities of food (including non-alcoholic beverages) people select and consume in the immediate and short term. However, it is uncertain whether reducing portions at the smaller end of the size range can be as effective in reducing food consumption as reductions at the larger end of the range. We are unable to highlight clear implications for tobacco or alcohol policy due to identified gaps in the current evidence base.