Consensus-based guidance for conducting and reporting multi-analyst studies

Participatory action research to develop and implement multicomponent, multilevel strategies for implementing colorectal cancer screening interventions in American Indian communities in New Mexico

BACKGROUND

Despite the effectiveness of colorectal cancer (CRC) screening, American Indians (AIs) have low screening rates in the US. Many AIs receive care at Indian Health Services, Tribal, and Urban Indian (I/T/U) healthcare facilities, where published evidence regarding the implementation of CRC screening interventions is lacking. To address this gap, the University of New Mexico Comprehensive Cancer Center and the Albuquerque Area Southwest Tribal Epidemiology Center collaborated with two tribally-operated healthcare facilities in New Mexico with the goal of improving CRC screening rates among New Mexico's AI communities.

METHODS

Guided by the principles of Community Based Participatory Research, we engaged providers from the two tribal healthcare facilities and tribal community members through focus group (two focus groups with providers (n = 15) and four focus group and listening sessions with community members (n = 65)), to elicit perspectives on the feasibility and appropriateness of implementing The Guide to Community Preventive Services (The Community Guide) recommended evidence-based interventions (EBIs) and strategies for increasing CRC screening. Within each tribal healthcare facility, we engaged a Multisector Action Team (MAT) that participated in an implementation survey to document the extent to which their healthcare facilities were implementing EBIs and strategies, and an organizational readiness survey that queried whether their healthcare facilities could implement additional strategies to improve uptake of CRC screening.

RESULTS

The Community Guide recommended EBIs and strategies that received the most support as feasible and appropriate from community members included: one-on-one education from providers, reminders, small media, and interventions that reduced structural barriers. From the providers' perspective, feasible and acceptable strategies included one-on-one education, patient and provider reminders, and provider assessment and feedback. Universally, providers mentioned the need for patient navigators who could provide culturally appropriate education about CRC and assist with transportation, and improved support for coordinating clinical follow-up after screening. The readiness survey highlighted overall readiness of the tribal facility, while the implementation survey highlighted that few strategies were being implemented.

CONCLUSIONS

Findings from this study contribute to the limited literature around implementation research at tribal healthcare facilities and informed the selection of specific implementation strategies to promote the uptake of CRC screening in AI communities.

Raising awareness of uncertain choices in empirical data analysis: A teaching concept toward replicable research practices

Throughout their education and when reading the scientific literature, students may get the impression that there is a unique and correct analysis strategy for every data analysis task and that this analysis strategy will always yield a significant and noteworthy result. This expectation conflicts with a growing realization that there is a multiplicity of possible analysis strategies in empirical research, which will lead to overoptimism and nonreplicable research findings if it is combined with result-dependent selective reporting. Here, we argue that students are often ill-equipped for real-world data analysis tasks and unprepared for the dangers of selectively reporting the most promising results. We present a seminar course intended for advanced undergraduates and beginning graduate students of data analysis fields such as statistics, data science, or bioinformatics that aims to increase the awareness of uncertain choices in the analysis of empirical data and present ways to deal with these choices through theoretical modules and practical hands-on sessions.

EEGManyPipelines: A Large-scale, Grassroots Multi-analyst Study of Electroencephalography Analysis Practices in the Wild

The ongoing reproducibility crisis in psychology and cognitive neuroscience has sparked increasing calls to re-evaluate and reshape scientific culture and practices. Heeding those calls, we have recently launched the EEGManyPipelines project as a means to assess the robustness of EEG research in naturalistic conditions and experiment with an alternative model of conducting scientific research. One hundred sixty-eight analyst teams, encompassing 396 individual researchers from 37 countries, independently analyzed the same unpublished, representative EEG data set to test the same set of predefined hypotheses and then provided their analysis pipelines and reported outcomes. Here, we lay out how large-scale scientific projects can be set up in a grassroots, community-driven manner without a central organizing laboratory. We explain our recruitment strategy, our guidance for analysts, the eventual outputs of this project, and how it might have a lasting impact on the field.

Excess death estimates from multiverse analysis in 2009-2021

Excess death estimates have great value in public health, but they can be sensitive to analytical choices. Here we propose a multiverse analysis approach that considers all possible different time periods for defining the reference baseline and a range of 1 to 4 years for the projected time period for which excess deaths are calculated. We used data from the Human Mortality Database on 33 countries with detailed age-stratified death information on an annual basis during the period 2009-2021. The use of different time periods for reference baseline led to large variability in the absolute magnitude of the exact excess death estimates. However, the relative ranking of different countries compared to others for specific years remained largely unaltered. The relative ranking of different years for the specific country was also largely independent of baseline. Averaging across all possible analyses, distinct time patterns were discerned across different countries. Countries had declines between 2009 and 2019, but the steepness of the decline varied markedly. There were also large differences across countries on whether the COVID-19 pandemic years 2020-2021 resulted in an increase of excess deaths and by how much. Consideration of longer projected time windows resulted in substantial shrinking of the excess deaths in many, but not all countries. Multiverse analysis of excess deaths over long periods of interest can offer an approach that better accounts for the uncertainty in estimating expected mortality patterns, comparative mortality trends across different countries, and the nature of observed mortality peaks.

Reproducibility in Neuroimaging Analysis: Challenges and Solutions

Recent years have marked a renaissance in efforts to increase research reproducibility in psychology, neuroscience, and related fields. Reproducibility is the cornerstone of a solid foundation of fundamental research-one that will support new theories built on valid findings and technological innovation that works. The increased focus on reproducibility has made the barriers to it increasingly apparent, along with the development of new tools and practices to overcome these barriers. Here, we review challenges, solutions, and emerging best practices with a particular emphasis on neuroimaging studies. We distinguish 3 main types of reproducibility, discussing each in turn. Analytical reproducibility is the ability to reproduce findings using the same data and methods. Replicability is the ability to find an effect in new datasets, using the same or similar methods. Finally, robustness to analytical variability refers to the ability to identify a finding consistently across variation in methods. The incorporation of these tools and practices will result in more reproducible, replicable, and robust psychological and brain research and a stronger scientific foundation across fields of inquiry.

How to build up big team science: a practical guide for large-scale collaborations

The past decade has witnessed a proliferation of big team science (BTS), endeavours where a comparatively large number of researchers pool their intellectual and/or material resources in pursuit of a common goal. Despite this burgeoning interest, there exists little guidance on how to create, manage and participate in these collaborations. In this paper, we integrate insights from a multi-disciplinary set of BTS initiatives to provide a how-to guide for BTS. We first discuss initial considerations for launching a BTS project, such as building the team, identifying leadership, governance, tools and open science approaches. We then turn to issues related to running and completing a BTS project, such as study design, ethical approvals and issues related to data collection, management and analysis. Finally, we address topics that present special challenges for BTS, including authorship decisions, collaborative writing and team decision-making.

One data set, many analysts: Implications for practicing scientists

Researchers routinely face choices throughout the data analysis process. It is often opaque to readers how these choices are made, how they affect the findings, and whether or not data analysis results are unduly influenced by subjective decisions. This concern is spurring numerous investigations into the variability of data analysis results. The findings demonstrate that different teams analyzing the same data may reach different conclusions. This is the "many-analysts" problem. Previous research on the many-analysts problem focused on demonstrating its existence, without identifying specific practices for solving it. We address this gap by identifying three pitfalls that have contributed to the variability observed in many-analysts publications and providing suggestions on how to avoid them.

The thresholding problem and variability in the EEG graph network parameters

Graph thresholding is a frequently used practice of eliminating the weak connections in brain functional connectivity graphs. The main aim of the procedure is to delete the spurious connections in the data. However, the choice of the threshold is arbitrary, and the effect of the threshold choice is not fully understood. Here we present the description of the changes in the global measures of a functional connectivity graph depending on the different proportional thresholds based on the 146 resting-state EEG recordings. The dynamics is presented in five different synchronization measures (wPLI, ImCoh, Coherence, ciPLV, PPC) in sensors and source spaces. The analysis shows significant changes in the graph's global connectivity measures as a function of the chosen threshold which may influence the outcome of the study. The choice of the threshold could lead to different study conclusions; thus it is necessary to improve the reasoning behind the choice of the different analytic options and consider the adoption of different analytic approaches. We also proposed some ways of improving the procedure of thresholding in functional connectivity research.

Questioning the vulnerability model: Prospective associations between low self-esteem and subsequent depression ratings may be spurious

BACKGROUND

According to the vulnerability model, low self-esteem makes people more depressed. Support for the vulnerability model comes almost exclusively from analyses using cross-lagged panel models, showing a negative effect of initial self-esteem on subsequent depression ratings when adjusting for initial depression. However, it is well known that such adjusted effects are susceptible to regression toward the mean.

METHODS

Data from four waves of measurements in five different samples (total N = 2703) were analyzed with two different cross-lagged panel models, two different random intercept cross-lagged panel models, and two different latent change score models, predicting change forwards as well as backwards in time.

RESULTS

High initial self-esteem predicted both decreased and increased depression ratings between measurements and an increase in self-esteem between measurements predicted a concurrent decrease in depression ratings.

LIMITATIONS

Only data from two western countries, Switzerland and USA, were analyzed. Whether the main finding, that a prospective effect of self-esteem on subsequent depression ratings might be spurious, applies to other countries and cultures remains an open question.

CONCLUSIONS

Due to the incongruent results, any causal effect of self-esteem on depression ratings, and thus the vulnerability model as such, cannot be corroborated by the data and models analyzed here. Instead, we propose, tentatively, that prospective associations between self-esteem and depression ratings may be spurious due to a combination of reasons, including regression toward the mean. The indication that depression might not be affected by measures to improve individuals' self-esteem is of clinical relevance.