Reevaluating excess success in psychological science

Visual discrimination accuracy does not differ between nasal inhalation and exhalation when stimuli are voluntarily aligned to breathing phase

This study investigated the possible enhancement of visual discrimination accuracy by voluntarily adjusting the timing of stimulus presentation to a specific respiratory phase. Previous research has suggested that respiratory phases modulate perceptual and cognitive processing. For instance, a fearful face was identified faster when presented during nasal inhalation than during nasal exhalation, which could be related to changes in neural oscillatory activity synchronized with breathing in through one's nose. Based on such findings, the present study asked 40 young adults to perform an emotional discrimination task consisting of distinguishing fearful vs. neutral faces and a physical discrimination task consisting of distinguishing high- vs. low-contrast Gabor patches during nasal respiration. Participants presented themselves with the stimuli to be judged in a designated respiratory phase by pressing a button. It was hypothesized that fear discrimination accuracy would be higher during inhalation than exhalation if sensitivity to emotional stimuli increased during inhalation. Conversely, if overall visual sensitivity was enhanced during inhalation, the identical effect was expected for contrast discrimination. The results indicated that discrimination accuracy did not differ between inhalation and exhalation phases in either task. This result provided no evidence that the respiratory phase affected visual discrimination accuracy when people adjusted the timing of stimulus presentation to the onset of inhalation or exhalation.

Implications of "Too Good to Be True" for Replication, Theoretical Claims, and Experimental Design: An Example Using Prominent Studies of Racial Bias

In response to concerns about the validity of empirical findings in psychology, some scientists use replication studies as a way to validate good science and to identify poor science. Such efforts are resource intensive and are sometimes controversial (with accusations of researcher incompetence) when a replication fails to show a previous result. An alternative approach is to examine the statistical properties of the reported literature to identify some cases of poor science. This review discusses some details of this process for prominent findings about racial bias, where a set of studies seems “too good to be true.” This kind of analysis is based on the original studies, so it avoids criticism from the original authors about the validity of replication studies. The analysis is also much easier to perform than a new empirical study. A variation of the analysis can also be used to explore whether it makes sense to run a replication study. As demonstrated here, there are situations where the existing data suggest that a direct replication of a set of studies is not worth the effort. Such a conclusion should motivate scientists to generate alternative experimental designs that better test theoretical ideas.

Confirming the appearance of excess success: Reply to van Boxtel and Koch (2016)

van Boxtel and Koch (Psychonomic Bulletin & Review. doi: 10.3758/s13423-016-1010-0 , 2016) reported finding problems in the Test for Excess Success (TES) analysis in Francis (Psychonomic Bulletin & Review, 21, 1180-1187, 2014). They argued that their findings undermined the general analysis and the conclusions of the specific TES analysis for their article (van Boxtel & Koch in Psychological Science, 23(4), 410-418, 2012). As shown in this paper, their reported problems reflect misunderstandings about both the general properties of a TES analysis and how it was applied to their specific set of findings. Another look at the findings and theoretical claims in van Boxtel and Koch (Psychological Science, 23(4), 410-418, 2012) confirms the appearance of excess success.