1
|
Grabman JH, Dobbins IG, Dodson CS. Comparing human evaluations of eyewitness statements to a machine learning classifier under pristine and suboptimal lineup administration procedures. Cognition 2024; 251:105876. [PMID: 39004009 DOI: 10.1016/j.cognition.2024.105876] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/11/2024] [Revised: 06/20/2024] [Accepted: 07/03/2024] [Indexed: 07/16/2024]
Abstract
Recent work highlights the ability of verbal machine learning classifiers to distinguish between accurate and inaccurate recognition memory decisions (Dobbins, 2022; Dobbins & Kantner, 2019; Seale-Carlisle, Grabman, & Dodson, 2022). Given the surge of interest in these modeling techniques, there is an urgent need to investigate verbal classifiers' limitations - particularly in applied contexts such as when police collect eyewitness's confidence statements. We find that confirmatory feedback (e.g., "This study now has a total of 87 participants, 84 of them made the same decision as you!") weakens the relationship between identification accuracy and verbal classifier scores to a similar degree as mock witnesses' numeric confidence judgments (Experiment 1). Crucially, for the first time, we compare the discriminative value of verbal classifier scores to the ratings of human evaluators who assessed the identical verbal confidence statements (Experiment 2). Our results suggest that human evaluators outperform the classifier when mock witnesses received no feedback; however, the classifier matches (or exceeds) the performance of human evaluators when mock witnesses received confirmatory feedback. Providing lineup information to human evaluators resulted in a worse ability to distinguish between correct and filler identifications, suggesting that this particular information may encourage the use of inappropriate heuristics when rendering accuracy judgments. Overall, these results suggest that the utility of verbal classifiers may be enhanced when contextual effects (e.g., lineup presence) impair human estimates of others' performance, but that translating witnesses' statements into classifier scores will not fix the problems of an improperly conducted lineup procedure.
Collapse
Affiliation(s)
- Jesse H Grabman
- Department of Psychology, New Mexico State University, United States.
| | - Ian G Dobbins
- Department of Psychological & Brain Sciences, Washington University in Saint Louis, United States
| | - Chad S Dodson
- Department of Psychology, University of Virginia, United States
| |
Collapse
|
2
|
Adl Zarrabi A, Jeulin M, Bardet P, Commère P, Naccache L, Aucouturier JJ, Ponsot E, Villain M. A simple psychophysical procedure separates representational and noise components in impairments of speech prosody perception after right-hemisphere stroke. Sci Rep 2024; 14:15194. [PMID: 38956187 PMCID: PMC11219855 DOI: 10.1038/s41598-024-64295-y] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/19/2023] [Accepted: 06/06/2024] [Indexed: 07/04/2024] Open
Abstract
After a right hemisphere stroke, more than half of the patients are impaired in their capacity to produce or comprehend speech prosody. Yet, and despite its social-cognitive consequences for patients, aprosodia following stroke has received scant attention. In this report, we introduce a novel, simple psychophysical procedure which, by combining systematic digital manipulations of speech stimuli and reverse-correlation analysis, allows estimating the internal sensory representations that subtend how individual patients perceive speech prosody, and the level of internal noise that govern behavioral variability in how patients apply these representations. Tested on a sample of N = 22 right-hemisphere stroke survivors and N = 21 age-matched controls, the representation + noise model provides a promising alternative to the clinical gold standard for evaluating aprosodia (MEC): both parameters strongly associate with receptive, and not expressive, aprosodia measured by MEC within the patient group; they have better sensitivity than MEC for separating high-functioning patients from controls; and have good specificity with respect to non-prosody-related impairments of auditory attention and processing. Taken together, individual differences in either internal representation, internal noise, or both, paint a potent portrait of the variety of sensory/cognitive mechanisms that can explain impairments of prosody processing after stroke.
Collapse
Affiliation(s)
- Aynaz Adl Zarrabi
- Université de Franche-Comté, SUPMICROTECH, CNRS, Institut FEMTO-ST, 25000, Besançon, France
| | - Mélissa Jeulin
- Department of Physical Medicine & Rehabilitation, APHP/Hôpital Pitié-Salpêtrière, 75013, Paris, France
| | - Pauline Bardet
- Department of Physical Medicine & Rehabilitation, APHP/Hôpital Pitié-Salpêtrière, 75013, Paris, France
| | - Pauline Commère
- Department of Physical Medicine & Rehabilitation, APHP/Hôpital Pitié-Salpêtrière, 75013, Paris, France
| | - Lionel Naccache
- Department of Physical Medicine & Rehabilitation, APHP/Hôpital Pitié-Salpêtrière, 75013, Paris, France
- Paris Brain Institute (ICM), Inserm, CNRS, PICNIC-Lab, 75013, Paris, France
| | | | - Emmanuel Ponsot
- Science & Technology of Music and Sound, IRCAM/CNRS/Sorbonne Université, 75004, Paris, France
| | - Marie Villain
- Department of Physical Medicine & Rehabilitation, APHP/Hôpital Pitié-Salpêtrière, 75013, Paris, France.
- Paris Brain Institute (ICM), Inserm, CNRS, PICNIC-Lab, 75013, Paris, France.
| |
Collapse
|
3
|
Bruder C, Poeppel D, Larrouy-Maestri P. Perceptual (but not acoustic) features predict singing voice preferences. Sci Rep 2024; 14:8977. [PMID: 38637516 PMCID: PMC11026466 DOI: 10.1038/s41598-024-58924-9] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/07/2023] [Accepted: 04/03/2024] [Indexed: 04/20/2024] Open
Abstract
Why do we prefer some singers to others? We investigated how much singing voice preferences can be traced back to objective features of the stimuli. To do so, we asked participants to rate short excerpts of singing performances in terms of how much they liked them as well as in terms of 10 perceptual attributes (e.g.: pitch accuracy, tempo, breathiness). We modeled liking ratings based on these perceptual ratings, as well as based on acoustic features and low-level features derived from Music Information Retrieval (MIR). Mean liking ratings for each stimulus were highly correlated between Experiments 1 (online, US-based participants) and 2 (in the lab, German participants), suggesting a role for attributes of the stimuli in grounding average preferences. We show that acoustic and MIR features barely explain any variance in liking ratings; in contrast, perceptual features of the voices achieved around 43% of prediction. Inter-rater agreement in liking and perceptual ratings was low, indicating substantial (and unsurprising) individual differences in participants' preferences and perception of the stimuli. Our results indicate that singing voice preferences are not grounded in acoustic attributes of the voices per se, but in how these features are perceptually interpreted by listeners.
Collapse
Affiliation(s)
- Camila Bruder
- Max Planck Institute for Empirical Aesthetics, Frankfurt am Main, Germany.
| | - David Poeppel
- New York University, New York, NY, USA
- Ernst Strüngmann Institute for Neuroscience, Frankfurt, Germany
- Max Planck-NYU Center for Language, Music, and Emotion (CLaME), New York, USA
| | - Pauline Larrouy-Maestri
- Max Planck Institute for Empirical Aesthetics, Frankfurt am Main, Germany
- Max Planck-NYU Center for Language, Music, and Emotion (CLaME), New York, USA
| |
Collapse
|
4
|
Larrouy-Maestri P, Poeppel D, Pell MD. The Sound of Emotional Prosody: Nearly 3 Decades of Research and Future Directions. PERSPECTIVES ON PSYCHOLOGICAL SCIENCE 2024:17456916231217722. [PMID: 38232303 DOI: 10.1177/17456916231217722] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/19/2024]
Abstract
Emotional voices attract considerable attention. A search on any browser using "emotional prosody" as a key phrase leads to more than a million entries. Such interest is evident in the scientific literature as well; readers are reminded in the introductory paragraphs of countless articles of the great importance of prosody and that listeners easily infer the emotional state of speakers through acoustic information. However, despite decades of research on this topic and important achievements, the mapping between acoustics and emotional states is still unclear. In this article, we chart the rich literature on emotional prosody for both newcomers to the field and researchers seeking updates. We also summarize problems revealed by a sample of the literature of the last decades and propose concrete research directions for addressing them, ultimately to satisfy the need for more mechanistic knowledge of emotional prosody.
Collapse
Affiliation(s)
- Pauline Larrouy-Maestri
- Max Planck Institute for Empirical Aesthetics, Frankfurt, Germany
- School of Communication Sciences and Disorders, McGill University
- Max Planck-NYU Center for Language, Music, and Emotion, New York, New York
| | - David Poeppel
- Max Planck-NYU Center for Language, Music, and Emotion, New York, New York
- Department of Psychology and Center for Neural Science, New York University
- Ernst Strüngmann Institute for Neuroscience, Frankfurt, Germany
| | - Marc D Pell
- School of Communication Sciences and Disorders, McGill University
- Centre for Research on Brain, Language, and Music, Montreal, Quebec, Canada
| |
Collapse
|
5
|
Osses A, Spinelli E, Meunier F, Gaudrain E, Varnet L. Prosodic cues to word boundaries in a segmentation task assessed using reverse correlation. JASA EXPRESS LETTERS 2023; 3:095205. [PMID: 37756550 DOI: 10.1121/10.0021022] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 06/08/2023] [Accepted: 09/03/2023] [Indexed: 09/29/2023]
Abstract
When listening to speech sounds, listeners are able to exploit acoustic features that mark the boundaries between successive words, the so-called segmentation cues. These cues are typically investigated by directly manipulating features that are hypothetically related to segmentation. The current study uses a different approach based on reverse correlation, where the stimulus manipulations are based on minimal assumptions. The method was evaluated using pairs of phonemically identical sentences in French, whose prosody was changed by introducing random f0 trajectories and segment durations. Our results support a prominent perceptual role of the f0 rise and vowel duration at the beginning of content words.
Collapse
Affiliation(s)
- Alejandro Osses
- Laboratoire des Systèmes Perceptifs, Département d'Études Cognitives, École Normale Supérieure, PSL University, CNRS, Paris, France
| | - Elsa Spinelli
- Laboratoire de Psychologie et NeuroCognition, Université Grenoble Alpes, Grenoble, France
| | | | - Etienne Gaudrain
- Lyon Neuroscience Research Center, CNRS, Inserm, Université Lyon 1, Lyon, , , , ,
| | - Léo Varnet
- Laboratoire des Systèmes Perceptifs, Département d'Études Cognitives, École Normale Supérieure, PSL University, CNRS, Paris, France
| |
Collapse
|
6
|
Ji Y, Hu Y, Jiang X. Segmental and suprasegmental encoding of speaker confidence in Wuxi dialect vowels. Front Psychol 2022; 13:1028106. [PMID: 36578688 PMCID: PMC9791101 DOI: 10.3389/fpsyg.2022.1028106] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/25/2022] [Accepted: 11/15/2022] [Indexed: 12/14/2022] Open
Abstract
Introduction Wuxi dialect is a variation of Wu dialect spoken in eastern China and is characterized by a rich tonal system. Compared with standard Mandarin speakers, those of Wuxi dialect as their mother tongue can be more efficient in varying vocal cues to encode communicative meanings in speech communication. While literature has demonstrated that speakers encode high vs. low confidence in global prosodic cues at the sentence level, it is unknown how speakers' intended confidence is encoded at a more local, phonetic level. This study aimed to explore the effects of speakers' intended confidence on both prosodic and formant features of vowels in two lexical tones (the flat tone and the contour tone) of Wuxi dialect. Methods Words of a single vowel were spoken in confident, unconfident, or neutral tone of voice by native Wuxi dialect speakers using a standard elicitation procedure. Linear-mixed effects modeling and parametric bootstrapping testing were performed. Results The results showed that (1) the speakers raised both F1 and F2 in the confident level (compared with the neutral-intending expression). Additionally, F1 can distinguish between the confident and unconfident expressions; (2) Compared with the neutral-intending expression, the speakers raised mean f0, had a greater variation of f0 and prolonged pronunciation time in the unconfident level while they raised mean intensity, had a greater variation of intensity and prolonged pronunciation time in the confident level. (3) The speakers modulated mean f0 and mean intensity to a larger extent on the flat tone than the contour tone to differentiate between levels of confidence in the voice, while they modulated f0 and intensity range more only on the contour tone. Discussion These findings shed new light on the mechanisms of segmental and suprasegmental encoding of speaker confidence and lack of confidence at the vowel level, highlighting the interplay of lexical tone and vocal expression in speech communication.
Collapse
|
7
|
Guerouaou N, Vaiva G, Aucouturier JJ. The shallow of your smile: the ethics of expressive vocal deep-fakes. Philos Trans R Soc Lond B Biol Sci 2022; 377:20210083. [PMID: 34775820 PMCID: PMC8591385 DOI: 10.1098/rstb.2021.0083] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/31/2021] [Accepted: 07/28/2021] [Indexed: 11/12/2022] Open
Abstract
Rapid technological advances in artificial intelligence are creating opportunities for real-time algorithmic modulations of a person's facial and vocal expressions, or 'deep-fakes'. These developments raise unprecedented societal and ethical questions which, despite much recent public awareness, are still poorly understood from the point of view of moral psychology. We report here on an experimental ethics study conducted on a sample of N = 303 participants (predominantly young, western and educated), who evaluated the acceptability of vignettes describing potential applications of expressive voice transformation technology. We found that vocal deep-fakes were generally well accepted in the population, notably in a therapeutic context and for emotions judged otherwise difficult to control, and surprisingly, even if the user lies to their interlocutors about using them. Unlike other emerging technologies like autonomous vehicles, there was no evidence of social dilemma in which one would, for example, accept for others what they resent for themselves. The only real obstacle to the massive deployment of vocal deep-fakes appears to be situations where they are applied to a speaker without their knowing, but even the acceptability of such situations was modulated by individual differences in moral values and attitude towards science fiction. This article is part of the theme issue 'Voice modulation: from origin and mechanism to social impact (Part II)'.
Collapse
Affiliation(s)
- Nadia Guerouaou
- Science and Technology of Music and Sound, IRCAM/CNRS/Sorbonne Université, Paris, France
- Lille Neuroscience and Cognition Center (LiNC), Team PSY, INSERM U-1172/CHRU Lille, France
| | - Guillaume Vaiva
- Lille Neuroscience and Cognition Center (LiNC), Team PSY, INSERM U-1172/CHRU Lille, France
| | | |
Collapse
|
8
|
Pruvost-Robieux E, André-Obadia N, Marchi A, Sharshar T, Liuni M, Gavaret M, Aucouturier JJ. It’s not what you say, it’s how you say it: a retrospective study of the impact of prosody on own-name P300 in comatose patients. Clin Neurophysiol 2022; 135:154-161. [DOI: 10.1016/j.clinph.2021.12.015] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/13/2021] [Revised: 12/16/2021] [Accepted: 12/18/2021] [Indexed: 02/05/2023]
|
9
|
Goupil L, Aucouturier JJ. Distinct signatures of subjective confidence and objective accuracy in speech prosody. Cognition 2021; 212:104661. [PMID: 33756151 DOI: 10.1016/j.cognition.2021.104661] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/21/2020] [Revised: 11/02/2020] [Accepted: 03/07/2021] [Indexed: 10/21/2022]
Abstract
Whether speech prosody truly and naturally reflects a speaker's subjective confidence, rather than other dimensions such as objective accuracy, is unclear. Here, using a new approach combing psychophysics with acoustic analysis and automatic classification of verbal reports, we tease apart the contributions of sensory evidence, accuracy, and subjective confidence to speech prosody. We find that subjective confidence and objective accuracy are distinctly reflected in the loudness, duration and intonation of verbal reports. Strikingly, we show that a speaker's accuracy is encoded in speech prosody beyond their own metacognitive awareness, and that it can be automatically decoded from this information alone with performances up to 60%. These findings demonstrate that confidence and accuracy have separable prosodic signatures that are manifested with different timings, and on different acoustic dimensions. Thus, both subjective mental states of confidence, and objective states related to competence, can be directly inferred from this natural behavior.
Collapse
Affiliation(s)
- Louise Goupil
- Laboratoire STMS, UMR 9912, CNRS/IRCAM/SU, Paris, France; University of East London, London, United Kingdom.
| | - Jean-Julien Aucouturier
- Laboratoire STMS, UMR 9912, CNRS/IRCAM/SU, Paris, France; FEMTO-ST, UMR 6174, CNRS/UBFC/ENSMM/UTBM, Besançon, France
| |
Collapse
|
10
|
Goupil L, Johansson P, Hall L, Aucouturier JJ. Vocal signals only impact speakers' own emotions when they are self-attributed. Conscious Cogn 2021; 88:103072. [PMID: 33406449 DOI: 10.1016/j.concog.2020.103072] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/28/2019] [Revised: 11/02/2020] [Accepted: 12/12/2020] [Indexed: 11/30/2022]
Abstract
Emotions are often accompanied by vocalizations whose acoustic features provide information about the physiological state of the speaker. Here, we ask if perceiving these affective signals in one's own voice has an impact on one's own emotional state, and if it is necessary to identify these signals as self-originated for the emotional effect to occur. Participants had to deliberate out loud about how they would feel in various familiar emotional scenarios, while we covertly manipulated their voices in order to make them sound happy or sad. Perceiving the artificial affective signals in their own voice altered participants' judgements about how they would feel in these situations. Crucially, this effect disappeared when participants detected the vocal manipulation, either explicitly or implicitly. The original valence of the scenarios also modulated the vocal feedback effect. These results highlight the role of the exteroception of self-attributed affective signals in the emergence of emotional feelings.
Collapse
Affiliation(s)
- Louise Goupil
- STMS UMR 9912 (CNRS/IRCAM/SU), Paris, France; University of East London, London, UK.
| | - Petter Johansson
- Lund University Cognitive Science, Lund University, Lund, Sweden
| | - Lars Hall
- Lund University Cognitive Science, Lund University, Lund, Sweden
| | | |
Collapse
|