Neurocomputational mechanisms underlying cross-modal associations and their influence on perceptual decisions

Inducing Novel Sound-Taste Correspondences via an Associative Learning Task

The interest in crossmodal correspondences, including those involving sounds and involving tastes, has experienced rapid growth in recent years. However, the mechanisms underlying these correspondences are not well understood. In the present study (N = 302), we used an associative learning paradigm, based on previous literature using simple sounds with no consensual taste associations (i.e., square and triangle wave sounds at 200 Hz) and taste words (i.e., sweet and bitter), to test the influence of two potential mechanisms in establishing sound-taste correspondences and investigate whether either learning mechanism could give rise to new and long-lasting associations. Specifically, we examined an emotional mediation account (i.e., using sad and happy emoji facial expressions) and a transitive path (i.e., sound-taste correspondence being mediated by color, using red and black colored squares). The results revealed that the associative learning paradigm mapping the triangle wave tone with a happy emoji facial expression induced a novel crossmodal correspondence between this sound and the word sweet. Importantly, we found that this novel association was still present two months after the experimental learning paradigm. None of the other mappings, emotional or transitive, gave rise to any significant associations between sound and taste. These findings provide evidence that new crossmodal correspondences between sounds and tastes can be created by leveraging the affective connection between both dimensions, helping elucidate the mechanisms underlying these associations. Moreover, these findings reveal that these associations can last for several weeks after the experimental session through which they were induced.

Attentional State Classification Using Amplitude and Phase Feature Extraction Method Based on Filter Bank and Riemannian Manifold

As a significant aspect of cognition, attention has been extensively studied and numerous measurements have been developed based on brain signal processing. Although existing attentional state classification methods have achieved good accuracy by extracting a variety of handcrafted features, spatial features have not been fully explored. This paper proposes an attentional state classification method based on Riemannian manifold to utilize spatial information. Based on the concept of Riemannian manifold of symmetric positive definite (SPD) matrix, the proposed method exploits the structure of covariance matrix to extract spatial features instead of using spatial filters. Specifically, Riemannian distances from intra-class Riemannian means are extracted as features for their robustness. To fully extend the potential of electroencephalograph (EEG) signal, both amplitude and phase information is utilized. In addition, to solve the variance of frequency bands, a filter bank is employed to process the signal of different frequency bands separately. Finally, features are fed into a support vector machine with a polynomial kernel to obtain classification results. The proposed attentional state classification using amplitude and phase feature extraction method based on filter bank and Riemannian manifold (AP-FBRM) method is evaluated on two open datasets including EEG data of 29 and 26 subjects. According to the experimental results, the optimal set of filter bank and the optimal technique to extract features containing both amplitude and phase information are determined. The proposed method respectively achieves accuracies of 88.06% and 80.00% and outperforms 8 baseline methods, which manifests that the proposed method creates an efficient way to recognize attentional state.