On the variability of dynamic functional connectivity assessment methods

BACKGROUND

Dynamic functional connectivity (dFC) has become an important measure for understanding brain function and as a potential biomarker. However, various methodologies have been developed for assessing dFC, and it is unclear how the choice of method affects the results. In this work, we aimed to study the results variability of commonly used dFC methods.

METHODS

We implemented 7 dFC assessment methods in Python and used them to analyze the functional magnetic resonance imaging data of 395 subjects from the Human Connectome Project. We measured the similarity of dFC results yielded by different methods using several metrics to quantify overall, temporal, spatial, and intersubject similarity.

RESULTS

Our results showed a range of weak to strong similarity between the results of different methods, indicating considerable overall variability. Somewhat surprisingly, the observed variability in dFC estimates was found to be comparable to the expected functional connectivity variation over time, emphasizing the impact of methodological choices on the final results. Our findings revealed 3 distinct groups of methods with significant intergroup variability, each exhibiting distinct assumptions and advantages.

CONCLUSIONS

Overall, our findings shed light on the impact of dFC assessment analytical flexibility and highlight the need for multianalysis approaches and careful method selection to capture the full range of dFC variation. They also emphasize the importance of distinguishing neural-driven dFC variations from physiological confounds and developing validation frameworks under a known ground truth. To facilitate such investigations, we provide an open-source Python toolbox, PydFC, which facilitates multianalysis dFC assessment, with the goal of enhancing the reliability and interpretability of dFC studies.

Observing many researchers using the same data and hypothesis reveals a hidden universe of uncertainty

This study explores how researchers' analytical choices affect the reliability of scientific findings. Most discussions of reliability problems in science focus on systematic biases. We broaden the lens to emphasize the idiosyncrasy of conscious and unconscious decisions that researchers make during data analysis. We coordinated 161 researchers in 73 research teams and observed their research decisions as they used the same data to independently test the same prominent social science hypothesis: that greater immigration reduces support for social policies among the public. In this typical case of social science research, research teams reported both widely diverging numerical findings and substantive conclusions despite identical start conditions. Researchers' expertise, prior beliefs, and expectations barely predict the wide variation in research outcomes. More than 95% of the total variance in numerical results remains unexplained even after qualitative coding of all identifiable decisions in each team's workflow. This reveals a universe of uncertainty that remains hidden when considering a single study in isolation. The idiosyncratic nature of how researchers' results and conclusions varied is a previously underappreciated explanation for why many scientific hypotheses remain contested. These results call for greater epistemic humility and clarity in reporting scientific findings.

Use of publication procedures to improve research integrity by addiction journals

BACKGROUND AND AIMS

The credibility crisis evident in many academic disciplines has led peer-reviewed journals to implement procedures to reduce use of flexible data analysis practices and selective reporting of results. This exploratory study examined the adoption of six of these procedures by addiction journals.

METHODS

Thirty-eight high-impact addiction journals were identified using the 2018 Clarivate Analytics Journal Citation Report for 2017 ranks. The online instructions for authors were reviewed for references to six publication procedures: conflict of interest disclosure, reporting guidelines, clinical trial registration, registration of other study designs, data-sharing and registered reports. The webpages of the Center for Open Science and Consolidated Standards of Reporting Trials (CONSORT) were also reviewed for data pertaining to registered reports and reporting guidelines, respectively.

RESULTS

The range of procedures adopted by the addiction journals was 0-5, with a mean of 2.66. Conflict-of-interest disclosure was required by all but one journal. Encouraging data-sharing was the next most commonly required procedure. Fewer than half the journals recommended specific reporting guidelines or required registration of clinical trials, and only four required procedures to pre-specify hypotheses and analytical methods.

CONCLUSIONS

While many addiction journals have adopted publication procedures to improve research integrity, these can be limited by their voluntary nature and monitoring difficulties. More stringent requirements that lock researchers into specific hypotheses and analyses have not been widely adopted.