1
|
Kelly SD, Ngo Tran QA. Exploring the Emotional Functions of Co-Speech Hand Gesture in Language and Communication. Top Cogn Sci 2023. [PMID: 37115518 DOI: 10.1111/tops.12657] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/07/2021] [Revised: 04/05/2023] [Accepted: 04/06/2023] [Indexed: 04/29/2023]
Abstract
Research over the past four decades has built a convincing case that co-speech hand gestures play a powerful role in human cognition . However, this recent focus on the cognitive function of gesture has, to a large extent, overlooked its emotional role-a role that was once central to research on bodily expression. In the present review, we first give a brief summary of the wealth of research demonstrating the cognitive function of co-speech gestures in language acquisition, learning, and thinking. Building on this foundation, we revisit the emotional function of gesture across a wide range of communicative contexts, from clinical to artistic to educational, and spanning diverse fields, from cognitive neuroscience to linguistics to affective science. Bridging the cognitive and emotional functions of gesture highlights promising avenues of research that have varied practical and theoretical implications for human-machine interactions, therapeutic interventions, language evolution, embodied cognition, and more.
Collapse
Affiliation(s)
- Spencer D Kelly
- Department of Psychological and Brain Sciences, Center for Language and Brain, Colgate University, 13 Oak Dr., Hamilton, NY, 13346, United States
| | - Quang-Anh Ngo Tran
- Department of Psychological and Brain Sciences, Indiana University, 1101 E. 10th St., Bloomington, IN, 47405, United States
| |
Collapse
|
2
|
Abdulmohsin HA, Al-Khateeb B, Hasan SS, Dwivedi R. Automatic illness prediction system through speech. COMPUTERS & ELECTRICAL ENGINEERING : AN INTERNATIONAL JOURNAL 2022; 102:108224. [PMID: 35880184 PMCID: PMC9302036 DOI: 10.1016/j.compeleceng.2022.108224] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 01/30/2022] [Revised: 06/29/2022] [Accepted: 07/08/2022] [Indexed: 06/15/2023]
Abstract
Due to the COVID-19 epidemic and the curfew caused by it, many people have sought to find an ADPS on the internet in the last few years. This hints to a new age of medical treatment, all the more so if the number of internet users continues to expand. As a result, automatic illness prediction online applications have attracted the interest of a large number of researchers worldwide. This work aims to develop and implement an automated illness prediction system based on speech. The system will be designed to forecast the sort of ailment a patient is suffering from based on his voice, but this was not feasible during the trial, therefore the diseases were divided into three categories (painful, light pain and psychological pain), and then the diagnose process were implemented accordingly. The medical dataset named "speech, transcription, and intent" served as the baseline for this study. The smoothness, MFCC, and SCV properties were used in this work, which demonstrated their high representation to human being medical situations. The noise reduction forward-backward filter was used to eliminate noise from wave files captured online in order to account for the high level of noise seen in the deployed dataset. For this study, a hybrid feature selection method was created and built that combined the output of a genetic algorithm (GA) with the inputs of a NN algorithm. Classification was performed using SVM, neural network, and GMM. The greatest results obtained were 94.55% illness classification accuracy in terms of SVM. The results showed that diagnosing illness through speech is a difficult process, especially when diagnosing each type of illness separately, but when grouping the different illness types into groups, depending on the amount of pain and the psychological situation of the patient, the results were much higher.
Collapse
Key Words
- ADPS, Automated Disease Prediction System
- Automatic disease prediction
- CPU, Central Processing Unit
- Forward-backward filter
- GA, Genetic Algorithm
- GB, Giga Byte
- GMM, Gaussian Mixture Model
- MFCC, Mel Frequency Cepstral Co-efficient
- Medical speech transcription and intent dataset
- Mel frequency Cepstral coefficient
- NN, Neural Network
- Neural network
- RAM, Random Access Memory
- RSM, Response Service Methodology
- SCV, Spectral Centroid Variability
- SVM, Support Vector Machine
- Spectral centroid variability
Collapse
Affiliation(s)
- Husam Ali Abdulmohsin
- Computer Science Department, College of Science, University of Baghdad, Baghdad, Iraq
| | - Belal Al-Khateeb
- College of Computer Science and information Technology, University of Anbar, Anbar, Iraq
| | - Samer Sami Hasan
- Computer Science Department, College of Science, University of Baghdad, Baghdad, Iraq
| | - Rinky Dwivedi
- Maharaja Surajmal Institute of Technology, New Delhi, India
| |
Collapse
|
3
|
Helmich I, Reinecke K, Meuter K, Simalla N, Ollinger N, Junge R, Lausberg H. Symptoms after sport-related concussions alter gestural functions. J Sci Med Sport 2020; 23:437-441. [DOI: 10.1016/j.jsams.2019.11.013] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/28/2019] [Revised: 09/27/2019] [Accepted: 11/27/2019] [Indexed: 10/25/2022]
|
4
|
Speed LJ, Majid A. Grounding language in the neglected senses of touch, taste, and smell. Cogn Neuropsychol 2019; 37:363-392. [DOI: 10.1080/02643294.2019.1623188] [Citation(s) in RCA: 12] [Impact Index Per Article: 2.4] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/26/2022]
Affiliation(s)
- Laura J. Speed
- Department of Psychology, University of York, York, England
| | - Asifa Majid
- Department of Psychology, University of York, York, England
| |
Collapse
|
5
|
Piraccini E, Maitan S. Pain communication during medical examination: beyond words. Minerva Anestesiol 2019; 85:1243-1244. [PMID: 31213048 DOI: 10.23736/s0375-9393.19.13784-4] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/08/2022]
Affiliation(s)
- Emanuele Piraccini
- Section of Anesthesia and Intensive Care, Department of Surgery, G.B. Morgagni-Pierantoni Hospital, Forlì, Forlì-Cesena, Italy -
| | - Stefano Maitan
- Section of Anesthesia and Intensive Care, Department of Surgery, G.B. Morgagni-Pierantoni Hospital, Forlì, Forlì-Cesena, Italy
| |
Collapse
|
6
|
Body-oriented gestures as a practitioner's window into interpreted communication. Soc Sci Med 2019; 233:171-180. [PMID: 31203145 DOI: 10.1016/j.socscimed.2019.05.040] [Citation(s) in RCA: 5] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/31/2018] [Revised: 05/21/2019] [Accepted: 05/24/2019] [Indexed: 11/22/2022]
Abstract
With increasing global migration, health care providers and patients may lack a shared language. Interpreters help to secure understanding. Doctors and patients cannot evaluate how the interpreter translates their utterances; however, they can see hand movements, which can provide a window into the interpretation process. While research on natural language use has acknowledged the semiotic contribution of co-speech gestures (i.e., spontaneous hand and arm movements that are tightly synchronized with speech), their role in interpreted interactions is unstudied. We aimed to reveal whether gestures could shed light on the interpreting process and to develop a systematic methodology for investigating gesture-use in interpreted encounters. Using data from authentic, interpreted clinical interactions, we identified and analyzed gestures referring to the body (i.e., body-oriented gestures). Data were 76 min of video-recorded doctor-patient consultations at two UK inner-city general practices in 2009. Using microanalysis of face-to-face dialogue, we revealed how participants used body-oriented gestures and how interpreters transmitted them. Participants used 264 body-oriented gestures (doctors = 113, patients = 54, interpreters = 97). Gestures served an important semiotic function: On average, 70% of the doctors' and patients' gestures provided information not conveyed in speech. When interpreters repeated the primary participants' body-oriented gestures, they were highly likely to accompany the gesture with speech that retained the overall utterance meaning. Conversely, when interpreters did not repeat the gesture, their speech tended to lack that information as well. A qualitative investigation into the local effect of gesture transmission suggested a means for quality control: visible discrepancies in interpretation generated opportunities to check understanding. The findings suggest that clinical communication training could benefit from including skills to understand and attend to gestures. The analysis developed here provides a promising schema and method for future research informing clinical guidelines and training.
Collapse
|
7
|
Tarr J, Cornish F, Gonzalez-Polledo E. Beyond the binaries: reshaping pain communication through arts workshops. SOCIOLOGY OF HEALTH & ILLNESS 2018; 40:577-592. [PMID: 29441591 DOI: 10.1111/1467-9566.12669] [Citation(s) in RCA: 3] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/08/2023]
Abstract
Pain is difficult to communicate and translate into language, yet most social research on pain experience uses questionnaires and semi-structured interviews that rely on words. In addition to the mind/body dualism prevalent in pain medicine in these studies pain communication is characterised by further value-laden binaries such as real/unreal, visible/invisible, and psychological/physical. Starting from the position that research methods play a role in constituting their object, this article examines the potential of participatory arts workshops for developing different versions of pain communication. Twenty-two participants were involved in workshops using drawing, digital photography, sound and physical theatre to explore pain communication. The use of arts materials made pain tangible. By manipulating pain-related objects, participants could consider alternative relationships to their pain. Pain's sociality was also explored, with relations with clinicians and others emerging as potentially cooperative rather than adversarial. Discussions considered whether pain felt internal or external, and whether it was possible to conceive of a self without pain. We argue that the socio-material context of participatory arts workshops enabled these alternative versions of pain. Such methods are a useful addition to medical sociology's heavy reliance on qualitative interviewing.
Collapse
Affiliation(s)
- Jen Tarr
- Department of Methodology, London School of Economics, UK
| | - Flora Cornish
- Department of Methodology, London School of Economics, UK
| | | |
Collapse
|
8
|
Rowbotham SJ, Holler J, Wearden A, Lloyd DM. I see how you feel: Recipients obtain additional information from speakers' gestures about pain. PATIENT EDUCATION AND COUNSELING 2016; 99:1333-1342. [PMID: 26996051 DOI: 10.1016/j.pec.2016.03.007] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 11/09/2015] [Revised: 03/07/2016] [Accepted: 03/13/2016] [Indexed: 06/05/2023]
Abstract
OBJECTIVE Despite the need for effective pain communication, pain is difficult to verbalise. Co-speech gestures frequently add information about pain that is not contained in the accompanying speech. We explored whether recipients can obtain additional information from gestures about the pain that is being described. METHODS Participants (n=135) viewed clips of pain descriptions under one of four conditions: 1) Speech Only; 2) Speech and Gesture; 3) Speech, Gesture and Face; and 4) Speech, Gesture and Face plus Instruction (short presentation explaining the pain information that gestures can depict). Participants provided free-text descriptions of the pain that had been described. Responses were scored for the amount of information obtained from the original clips. FINDINGS Participants in the Instruction condition obtained the most information, while those in the Speech Only condition obtained the least (all comparisons p<0.001). CONCLUSIONS Gestures produced during pain descriptions provide additional information about pain that recipients are able to pick up without detriment to their uptake of spoken information. PRACTICE IMPLICATIONS Healthcare professionals may benefit from instruction in gestures to enhance uptake of information about patients' pain experiences.
Collapse
Affiliation(s)
- Samantha J Rowbotham
- School of Psychological Sciences, University of Manchester, Manchester, UK; Menzies Centre for Health Policy, School of Public Health, University of Sydney, Sydney, Australia; The Australian Prevention Partnership Centre, Sax Institute, Sydney, Australia.
| | - Judith Holler
- Max Planck Institute for Psycholinguistics, Nijmegen, Netherlands.
| | - Alison Wearden
- School of Psychological Sciences, University of Manchester, Manchester, UK.
| | - Donna M Lloyd
- School of Psychology, University of Leeds, Leeds, UK.
| |
Collapse
|
9
|
Escher M, Büssing A, Ostermann T. Survey on hand gestures relevance in patient practitioner communication: a homeopathic example. HOMEOPATHY 2016; 105:233-239. [PMID: 27473544 DOI: 10.1016/j.homp.2016.02.005] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/20/2015] [Revised: 02/13/2016] [Accepted: 02/29/2016] [Indexed: 11/28/2022]
Abstract
BACKGROUND Gestures play an important role in medical communication. METHODS 94 homeopaths (Mean age 49.6 years, 20% male) completed a 20-item questionnaire on utilization and relevance of gestures in patients' symptom description. RESULTS After excluding nine items due to low validity (n = 4) or low item total correlation (n = 5), factor analysis of the questionnaire resulted in the following three dimensions explaining 66.6% of variance: 'Hand gestures in relation to verbal expressions' (5 items; Cronbach's α = 0.81), 'Hand Gestures describing the experience of bodily and mental symptoms' (4 items; Cronbach's α = 0.74) and 'practitioners' behavior and active attitude in observing hand gestures' (2 items; Cronbach's α = 0.86). CONCLUSION The survey shows how homeopathic therapists view patients' hand gestures, whether they use these diagnostically and how this relates to their homeopathic practice. Practitioners with only homeopathic influence on this topic are highly congruent to findings on hand gestures from other domains.
Collapse
Affiliation(s)
- Max Escher
- Institute for Integrative Medicine, Witten/Herdecke University, Gerhard-Kienle-Weg 4, 58313 Herdecke, Germany.
| | - Arndt Büssing
- Institute for Integrative Medicine, Witten/Herdecke University, Gerhard-Kienle-Weg 4, 58313 Herdecke, Germany
| | - Thomas Ostermann
- Institute for Integrative Medicine, Witten/Herdecke University, Gerhard-Kienle-Weg 4, 58313 Herdecke, Germany; Department of Clinical Psychology, Witten/Herdecke University, Alfred-Herrhausen-Str. 50, 58448 Witten, Germany
| |
Collapse
|
10
|
Rowbotham S, Wardy AJ, Lloyd DM, Wearden A, Holler J. Increased pain intensity is associated with greater verbal communication difficulty and increased production of speech and co-speech gestures. PLoS One 2014; 9:e110779. [PMID: 25343486 PMCID: PMC4208777 DOI: 10.1371/journal.pone.0110779] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.2] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/13/2014] [Accepted: 09/24/2014] [Indexed: 11/17/2022] Open
Abstract
Effective pain communication is essential if adequate treatment and support are to be provided. Pain communication is often multimodal, with sufferers utilising speech, nonverbal behaviours (such as facial expressions), and co-speech gestures (bodily movements, primarily of the hands and arms that accompany speech and can convey semantic information) to communicate their experience. Research suggests that the production of nonverbal pain behaviours is positively associated with pain intensity, but it is not known whether this is also the case for speech and co-speech gestures. The present study explored whether increased pain intensity is associated with greater speech and gesture production during face-to-face communication about acute, experimental pain. Participants (N = 26) were exposed to experimentally elicited pressure pain to the fingernail bed at high and low intensities and took part in video-recorded semi-structured interviews. Despite rating more intense pain as more difficult to communicate (t(25) = 2.21, p = .037), participants produced significantly longer verbal pain descriptions and more co-speech gestures in the high intensity pain condition (Words: t(25) = 3.57, p = .001; Gestures: t(25) = 3.66, p = .001). This suggests that spoken and gestural communication about pain is enhanced when pain is more intense. Thus, in addition to conveying detailed semantic information about pain, speech and co-speech gestures may provide a cue to pain intensity, with implications for the treatment and support received by pain sufferers. Future work should consider whether these findings are applicable within the context of clinical interactions about pain.
Collapse
Affiliation(s)
- Samantha Rowbotham
- School of Psychological Sciences, University of Manchester, Manchester, United Kingdom
| | - April J. Wardy
- School of Psychological Sciences, University of Manchester, Manchester, United Kingdom
| | - Donna M. Lloyd
- Institute of Psychological Sciences, University of Leeds, Leeds, United Kingdom
| | - Alison Wearden
- School of Psychological Sciences, University of Manchester, Manchester, United Kingdom
| | - Judith Holler
- Max Planck Institute for Psycholinguistics, Nijmegen, Netherlands
| |
Collapse
|
11
|
Gerwing J, Dalby AML. Gestures convey content: an exploration of the semantic functions of physicians' gestures. PATIENT EDUCATION AND COUNSELING 2014; 96:308-314. [PMID: 25103179 DOI: 10.1016/j.pec.2014.07.014] [Citation(s) in RCA: 7] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 01/16/2014] [Revised: 06/13/2014] [Accepted: 07/12/2014] [Indexed: 06/03/2023]
Abstract
OBJECTIVES Gestures' semiotic role in clinical interactions is unexplored. Using theoretical underpinnings from basic research on gesture, our objective was to investigate the semantic contributions of physicians' gestures during interactions with patients with a different native language. METHODS We analyzed gestures-speech composites in eight videotaped interactions between physicians and patients during treatment plan discussions. Using microanalysis of face-to-face dialogue and conversation analysis, we identified physicians' gestures, decided whether they served semantic functions, and explored their relationship with the accompanying speech. RESULTS Using the operational definitions developed here resulted in high reliability. Physicians gestured at a mean rate of 6.5 gestures per 100 words. Approximately half of the gestures served semantic functions, with referents that were concrete (e.g., actions, body parts) and abstract (e.g., regularity, timelines). Gestures conveyed topic information, but speech conveyed information about that topic and context for interpreting gestures' meaning. CONCLUSION Analyzing the semantic functions of gestures in clinical interactions is feasible. Physicians' gestures and speech formed integrated messages; the two modalities conveyed mutually dependent meanings. PRACTICE IMPLICATIONS Physicians could become aware of the semiotic potential of gestures. However, conversational gestures lack conventional meanings and rely on the accompanying speech to provide necessary context for interpreting their meaning.
Collapse
Affiliation(s)
- Jennifer Gerwing
- Akershus University Hospital, Health Service Research Centre, Lørenskog, Norway.
| | | |
Collapse
|
12
|
Stevenson F. Achieving visibility? Use of non-verbal communication in interactions between patients and pharmacists who do not share a common language. SOCIOLOGY OF HEALTH & ILLNESS 2014; 36:756-771. [PMID: 24641161 PMCID: PMC4285156 DOI: 10.1111/1467-9566.12102] [Citation(s) in RCA: 9] [Impact Index Per Article: 0.9] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Indexed: 06/03/2023]
Abstract
Despite the seemingly insatiable interest in healthcare professional-patient communication, less attention has been paid to the use of non-verbal communication in medical consultations. This article considers pharmacists' and patients' use of non-verbal communication to interact directly in consultations in which they do not share a common language. In total, 12 video-recorded, interpreted pharmacy consultations concerned with a newly prescribed medication or a change in medication were analysed in detail. The analysis focused on instances of direct communication initiated by either the patient or the pharmacist, despite the presence of a multilingual pharmacy assistant acting as an interpreter. Direct communication was shown to occur through (i) the demonstration of a medical device, (ii) the indication of relevant body parts and (iii) the use of limited English. These connections worked to make patients and pharmacists visible to each other and thus to maintain a sense of mutual involvement in consultations within which patients and pharmacists could enact professionally and socially appropriate roles. In a multicultural society this work is important in understanding the dynamics involved in consultations in situations in which language is not shared and thus in considering the development of future research and policy.
Collapse
Affiliation(s)
- Fiona Stevenson
- UCL Research Department of Primary Care and Population Health, University College LondonUK
| |
Collapse
|
13
|
Rowbotham S, Lloyd DM, Holler J, Wearden A. Externalizing the private experience of pain: a role for co-speech gestures in pain communication? HEALTH COMMUNICATION 2014; 30:70-80. [PMID: 24483213 DOI: 10.1080/10410236.2013.836070] [Citation(s) in RCA: 3] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/03/2023]
Abstract
Despite the importance of effective pain communication, talking about pain represents a major challenge for patients and clinicians because pain is a private and subjective experience. Focusing primarily on acute pain, this article considers the limitations of current methods of obtaining information about the sensory characteristics of pain and suggests that spontaneously produced "co-speech hand gestures" may constitute an important source of information here. Although this is a relatively new area of research, we present recent empirical evidence that reveals that co-speech gestures contain important information about pain that can both add to and clarify speech. Following this, we discuss how these findings might eventually lead to a greater understanding of the sensory characteristics of pain, and to improvements in treatment and support for pain sufferers. We hope that this article will stimulate further research and discussion of this previously overlooked dimension of pain communication.
Collapse
|