Rigor and replication: Toward improved best practices in human electrophysiology research

The future of psychophysiology, then and now

Since its founding in 1973, Biological Psychology has showcased and provided invaluable support to psychophysiology, a field that has grown and changed enormously. This article discusses some constancies that have remained fundamental to the journal and to the field as well as some important trends. Some aspects of our science have not received due consideration, affecting not only the generalizability of our findings but the way we develop and evaluate our research questions and the potential of our field to contribute to the common good. The article offers a number of predictions and recommendations for the next period of growth of psychophysiology.

Feedback negativity and feedback-related P3 in individuals at risk for depression: Comparing surface potentials and current source densities

Blunted responses to reward feedback have been linked to major depressive disorder (MDD) and depression risk. Using a monetary incentive delay task (win, loss, break-even), we investigated the impact of family risk for depression and lifetime history of MDD and anxiety disorder with 72-channel electroencephalograms (EEG) recorded from 29 high-risk and 32 low-risk individuals (15-58 years, 30 male). Linked-mastoid surface potentials (ERPs) and their corresponding reference-free current source densities (CSDs) were quantified by temporal principal components analysis (PCA). Each PCA solution revealed a midfrontal feedback negativity (FN; peak around 310 ms) and a posterior feedback-P3 (fb-P3; 380 ms) as two distinct reward processing stages. Unbiased permutation tests and multilevel modeling of component scores revealed greater FN to loss than win and neutral for all stratification groups, confirming FN sensitivity to valence. Likewise, all groups had greater fb-P3 to win and loss than neutral, confirming that fb-P3 indexes motivational salience and allocation of attention. By contrast, group effects were subtle, dependent on data transformation (ERP, CSD), and did not confirm reduced FN or fb-P3 for at-risk individuals. Instead, CSD-based fb-P3 was overall reduced in individuals with than without MDD history, whereas ERP-based fb-P3 was greater for high-risk individuals than for low-risk individuals for monetary, but not neutral outcomes. While the present findings do not support blunted reward processing in depression and depression risk, our side-by-side comparison underscores how the EEG reference choice affects the characterization of subtle group differences, strongly advocating the use of reference-free techniques.

Estimating statistical power for ERP studies using the auditory N1, Tb, and P2 components

The N1, Tb, and P2 components of the event-related potential (ERP) are thought to reflect the sequential processing of auditory stimuli in the human brain. Despite their extensive use in biological, cognitive, and clinical neuroscience, there are no guidelines for how to appropriately power ERP studies using these components. In the present study, we investigated how the number of trials, number of participants, effect magnitude, and study design influenced statistical power. Using Monte Carlo simulations of ERP data from a passive listening task, we determined the probability of finding a statistically significant effect in 58,900 experiments repeated 1,000 times each. We found that as the number of trials, number of participants, and effect magnitude increased, so did statistical power. We also found that increasing the number of trials had a bigger effect on statistical power for within-subject designs than for between-subject designs, and that within-subject designs required a smaller number of trials and participants to provide the same level of statistical power for a given effect magnitude than between-subject designs. These results show that it is important to carefully consider these factors when designing ERP studies, rather than relying on tradition or anecdotal evidence. To improve the robustness and reproducibility of ERP research, we have built an online statistical power calculator (https://bradleynjack.shinyapps.io/ErpPowerCalculator), which we hope will allow researchers to estimate the statistical power of previous studies, as well as help them design appropriately-powered studies in the future.

Systematically investigating the role of context on effect replicability in reinstatement of fear in humans

Context is crucial in guiding behavior in an ever-changing world and contextual information plays a crucial role in associative learning processes. For instance, the return of fear (RoF) after successful extinction, which is used to study the mechanisms underlying relapse phenomena in fear- and stress-related disorders in an experimental model, is known to be context dependent as evident from phenomena such as renewal (contextual change) and reinstatement (re-exposure to an aversive event). Human adaptions of reinstatement paradigms have resulted in mixed findings: CS specific as well as unspecific RoF or unexpected "reinstated" conditioned responding in no reinstatement US control groups. Here, we systematically investigate the role of context (i.e., cue-context compound) on reinstatement-induced RoF in a human differential fear conditioning paradigm using subjective and psychophysiological measures in a large sample (N = 212) including reinstatement and control groups. Overall, response patterns in reinstatement-groups mirrored results from single-cue rodent work. Yet, only generalized, not differential RoF was observed. Remarkably, depending on outcome measure RoF was also observed under identical experimental context conditions without US-re-exposure, underlining effects of contextual change beyond the reinstatement-US and challenging reinstatement research in human subjects and highlight that future reinstatement work should focus on the operationalization of context.

EPOS: EEG Processing Open-Source Scripts

Background

Since the replication crisis, standardization has become even more important in psychological science and neuroscience. As a result, many methods are being reconsidered, and researchers’ degrees of freedom in these methods are being discussed as a potential source of inconsistencies across studies.

New Method

With the aim of addressing these subjectivity issues, we have been working on a tutorial-like EEG (pre-)processing pipeline to achieve an automated method based on the semi-automated analysis proposed by Delorme and Makeig.

Results

Two scripts are presented and explained step-by-step to perform basic, informed ERP and frequency-domain analyses, including data export to statistical programs and visual representations of the data. The open-source software EEGlab in MATLAB is used as the data handling platform, but scripts based on code provided by Mike Cohen (2014) are also included.

Comparison with existing methods

This accompanying tutorial-like article explains and shows how the processing of our automated pipeline affects the data and addresses, especially beginners in EEG-analysis, as other (pre)-processing chains are mostly targeting rather informed users in specialized areas or only parts of a complete procedure. In this context, we compared our pipeline with a selection of existing approaches.

Conclusion

The need for standardization and replication is evident, yet it is equally important to control the plausibility of the suggested solution by data exploration. Here, we provide the community with a tool to enhance the understanding and capability of EEG-analysis. We aim to contribute to comprehensive and reliable analyses for neuro-scientific research.

Improving the Rigor and Replicability of Applied Psychophysiology Research: Sample Size, Standardization, Transparency, and Preregistration

Scientific research across a number of areas, including applied psychophysiology, biofeedback, and neurofeedback, is facing considerable scrutiny for poor replication rates, high numbers of false-positive findings, and insufficient scientific rigor. There are many factors underlying this replication crisis in scientific research; yet incentives for more rigorous research practices at the institutional and editorial levels lag behind the need for improvement. The author provides examples of replication and rigor difficulties in scientific research with an eye toward psychophysiological research, including researcher flexibility in data analysis, “p-hacking,” insufficient sample sizes, and lack of availability and implementation of rigorous methodological and publication guidelines. Subsequently, the author highlights examples and opportunities for improvement, including decreasing researcher flexibility, reporting sample size information, increasing sample sizes through collaboration, improving reporting standards/following established guidelines for reporting psychophysiological data, and increasing adoption of preregistration and registered reports. The author concludes that the Association for Applied Psychophysiology and Biofeedback (AAPB) can improve clinical practice and perception of public and scientific credibility by implementing rigorous and transparent research practices with a focus on replicability and clear methodological and reporting techniques and standards.

Experimental boundary conditions of reinstatement-induced return of fear in humans: Is reinstatement in humans what we think it is?

Experimental paradigms used to study reinstatement of fear in humans are characterized by procedural heterogeneity. Reinstatement protocols involve unexpected (re)-presentations of the unconditioned stimulus (USs) after fear extinction training. Here, we address the number of reinstatement USs administered as a potential boundary condition that may explain divergent findings in the field. A sample of 171 participants is exposed to a fear acquisition training, immediate extinction training, and reinstatement test experiment. Three groups differing in the number of reinstatement US are employed: one (n = 57) or four (n = 55) in experimental groups and zero (n = 59) in the control group. We adopt Bayesian statistical approaches beyond classical null hypothesis significance testing (NHST) to qualify evidence for or against this potential methodological boundary condition in reinstatement-induced return of fear. Startle potentiation to the reinstatement administration context was increased for the RI-US_one compared to the RI-US_zero group, supporting the role of context conditioning in reinstatement. This effect was weaker in the RI-US_four group. This, however, did not transfer to responding to conditioned stimuli during the return of fear-test: no evidence for an effect of the number of reinstatement USs (zero, one, four) was observed in behavioral or physiological measures. In sum, our results speak against the number of reinstatement USs as a potential boundary condition in experimentally induced return of fear in humans. This may challenge what we think we know about the reinstatement phenomenon in humans and call for critical reconsideration of paradigms as well as mechanisms that may underlie some reinstatement effects in the literature.

Integrated open-source software for multiscale electrophysiology

The methods for electrophysiology in neuroscience have evolved tremendously over the recent years with a growing emphasis on dense-array signal recordings. Such increased complexity and augmented wealth in the volume of data recorded, have not been accompanied by efforts to streamline and facilitate access to processing methods, which too are susceptible to grow in sophistication. Moreover, unsuccessful attempts to reproduce peer-reviewed publications indicate a problem of transparency in science. This growing problem could be tackled by unrestricted access to methods that promote research transparency and data sharing, ensuring the reproducibility of published results. Here, we provide a free, extensive, open-source software that provides data-analysis, data-management and multi-modality integration solutions for invasive neurophysiology. Users can perform their entire analysis through a user-friendly environment without the need of programming skills, in a tractable (logged) way. This work contributes to open-science, analysis standardization, transparency and reproducibility in invasive neurophysiology.

Quantifying evidential value and selective reporting in recent and 10-year past psychophysiological literature: A pre-registered P-curve analysis

Selective reporting (i.e., only reporting significant findings as opposed to all analyses or results) is a questionable research practice that undermines the integrity of published research. Psychophysiology research may be susceptible to selective reporting, given the high number of decision points and methodological complexity in analyses of psychophysiology data. We aimed to assess the presence of selective reporting and evidential value (i.e., that significant results are due to true underlying effects) in recent and past psychophysiological research by utilizing p-curve analyses. Study protocols and methods were pre-registered on the Open Science Framework (OSF). P-values and the associated test statistics were extracted from articles in the most recent issue (as of January 2018) and 10-year previous counterpart issue of three major psychophysiology journals: Psychophysiology, International Journal of Psychophysiology, and Journal of Psychophysiology. Using the p-curve application, 10 primary p-curves were conducted: all recent articles, all past articles, recent articles split by journal, past articles split by journal, recent cognitive electrophysiology articles, and past cognitive electrophysiology articles. Evidential value and generally adequate average power (≥78% average power) were present in all p-curves, except those that only included articles from the Journal of Psychophysiology because of the small number of articles published in the journal. Findings provide some positive news and indicate that, generally, results were not selectively reported, and selective reporting may not be a primary issue for this sample of psychophysiological research. Future p-curve analyses examining sub-disciplines of psychophysiology are recommended.

Do sport-related concussions result in long-term cognitive impairment? A review of event-related potential research

Sport-related concussions have become a major public health concern although the long-term effects on cognitive function remain largely unknown. Event-related potentials (ERPs) are ideal for studying the long-term impact of sport-related concussions, as they have excellent temporal precision and provide insight that cannot be obtained from behavioral or neuropsychological measures alone. We reviewed all available published studies that have used stimulus or response-locked ERPs to document cognitive control processes in individuals with a history of concussion. Collectively, cross-sectional evidence suggests consistent reductions in P3 amplitude in previously concussed individuals, as well as a possible impairment in cognitive processing speed (P3 latency) and error monitoring processes (ERN). The persistent neurophysiological changes found may be related to the number of previous concussions sustained and the time since injury. Future studies incorporating prospective research designs are warranted before definitive statements can be offered regarding the long-term impact of sport-related concussions on cognitive control.

Computational reproducibility of "Goal relevance and goal conduciveness appraisals lead to differential autonomic reactivity in emotional responding to performance feedback" (Kreibig, Gendolla, & Scherer, 2012): A guide and new evidence

The emerging field of the psychophysiology of motivation bears many new findings, but little replication. Using my own data (Kreibig, Gendolla, & Scherer, 2012), I test the reproducibility of this specific study, provide the necessary materials to make the study reproducible, and instantiate proper reproducibility practices that other researchers can use as a road map toward the same goal. In addition, based on re-analyses of the original data, I report new evidence for the motivational effects of emotional responding to performance feedback. Specifically, greater appraisal of goal relevance amplifies the emotional response to events appraised as conducive (i.e., effort mobilization), but not to those appraised as obstructive to a person's goals (i.e., effort withdrawal). I conclude by providing a ten-step road map of best practices to facilitate computational reproducibility for future studies.