1
|
Akiyama T, Matsumoto K, Osaka K, Tanioka R, Betriana F, Zhao Y, Kai Y, Miyagawa M, Yasuhara Y, Ito H, Soriano G, Tanioka T. Comparison of Subjective Facial Emotion Recognition and "Facial Emotion Recognition Based on Multi-Task Cascaded Convolutional Network Face Detection" between Patients with Schizophrenia and Healthy Participants. Healthcare (Basel) 2022; 10:healthcare10122363. [PMID: 36553887 PMCID: PMC9777528 DOI: 10.3390/healthcare10122363] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/13/2022] [Revised: 11/16/2022] [Accepted: 11/21/2022] [Indexed: 11/27/2022] Open
Abstract
Patients with schizophrenia may exhibit a flat affect and poor facial expressions. This study aimed to compare subjective facial emotion recognition (FER) and FER based on multi-task cascaded convolutional network (MTCNN) face detection in 31 patients with schizophrenia (patient group) and 40 healthy participants (healthy participant group). A Pepper Robot was used to converse with the 71 aforementioned participants; these conversations were recorded on video. Subjective FER (assigned by medical experts based on video recordings) and FER based on MTCNN face detection was used to understand facial expressions during conversations. This study confirmed the discriminant accuracy of the FER based on MTCNN face detection. The analysis of the smiles of healthy participants revealed that the kappa coefficients of subjective FER (by six examiners) and FER based on MTCNN face detection concurred (κ = 0.63). The perfect agreement rate between the subjective FER (by three medical experts) and FER based on MTCNN face detection in the patient, and healthy participant groups were analyzed using Fisher's exact probability test where no significant difference was observed (p = 0.72). The validity and reliability were assessed by comparing the subjective FER and FER based on MTCNN face detection. The reliability coefficient of FER based on MTCNN face detection was low for both the patient and healthy participant groups.
Collapse
Affiliation(s)
- Toshiya Akiyama
- Graduate School of Health Sciences, Tokushima University, Tokushima 770-8509, Japan
| | - Kazuyuki Matsumoto
- Graduate School of Engineering, Tokushima University, Tokushima 770-8506, Japan
| | - Kyoko Osaka
- Department of Psychiatric Nursing, Nursing Course of Kochi Medical School, Kochi University, Kochi 783-8505, Japan
| | - Ryuichi Tanioka
- Department of Physical Therapy, Hiroshima Cosmopolitan University, Hiroshima 734-0014, Japan
| | | | - Yueren Zhao
- Department of Psychiatry, Fujita Health University, Nagoya 470-1192, Japan
| | - Yoshihiro Kai
- Department of Mechanical Engineering, Tokai University, Tokyo 151-8677, Japan
| | - Misao Miyagawa
- Department of Nursing, Faculty of Health and Welfare, Tokushima Bunri University, Tokushima 770-8514, Japan
| | - Yuko Yasuhara
- Institute of Biomedical Sciences, Tokushima University, Tokushima 770-8509, Japan
| | - Hirokazu Ito
- Institute of Biomedical Sciences, Tokushima University, Tokushima 770-8509, Japan
| | - Gil Soriano
- Department of Nursing, College of Allied Health, National University Philippines, Manila 1008, Philippines
| | - Tetsuya Tanioka
- Institute of Biomedical Sciences, Tokushima University, Tokushima 770-8509, Japan
- Correspondence:
| |
Collapse
|
2
|
Osaka K, Matsumoto K, Akiyama T, Tanioka R, Betriana F, Zhao Y, Kai Y, Miyagawa M, Tanioka T, Locsin RC. Investigation of Methods to Create Future Multimodal Emotional Data for Robot Interactions in Patients with Schizophrenia: A Case Study. Healthcare (Basel) 2022; 10:healthcare10050848. [PMID: 35627984 PMCID: PMC9140390 DOI: 10.3390/healthcare10050848] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/02/2022] [Revised: 04/28/2022] [Accepted: 05/01/2022] [Indexed: 11/16/2022] Open
Abstract
Rapid progress in humanoid robot investigations offers possibilities for improving the competencies of people with social disorders, although this improvement of humanoid robots remains unexplored for schizophrenic people. Methods for creating future multimodal emotional data for robot interactions were studied in this case study of a 40-year-old male patient with disorganized schizophrenia without comorbidities. The qualitative data included heart rate variability (HRV), video-audio recordings, and field notes. HRV, Haar cascade classifier (HCC), and Empath API© were evaluated during conversations between the patient and robot. Two expert nurses and one psychiatrist evaluated facial expressions. The research hypothesis questioned whether HRV, HCC, and Empath API© are useful for creating future multimodal emotional data about robot–patient interactions. The HRV analysis showed persistent sympathetic dominance, matching the human–robot conversational situation. The result of HCC was in agreement with that of human observation, in the case of rough consensus. In the case of observed results disagreed upon by experts, the HCC result was also different. However, emotional assessments by experts using Empath API© were also found to be inconsistent. We believe that with further investigation, a clearer identification of methods for multimodal emotional data for robot interactions can be achieved for patients with schizophrenia.
Collapse
Affiliation(s)
- Kyoko Osaka
- Department of Psychiatric Nursing, Nursing Course of Kochi Medical School, Kochi University, Kochi 783-8505, Japan
- Correspondence:
| | - Kazuyuki Matsumoto
- Graduate School of Engineering, Tokushima University, Tokushima 770-8506, Japan;
| | - Toshiya Akiyama
- Graduate School of Health Sciences, Tokushima University, Tokushima 770-8509, Japan; (T.A.); (R.T.); (F.B.)
| | - Ryuichi Tanioka
- Graduate School of Health Sciences, Tokushima University, Tokushima 770-8509, Japan; (T.A.); (R.T.); (F.B.)
| | - Feni Betriana
- Graduate School of Health Sciences, Tokushima University, Tokushima 770-8509, Japan; (T.A.); (R.T.); (F.B.)
| | - Yueren Zhao
- Department of Psychiatry, School of Medicine, Fujita Health University, Toyoake 470-1192, Japan;
| | - Yoshihiro Kai
- Department of Mechanical Engineering, Tokai University, Tokyo 259-1292, Japan;
| | - Misao Miyagawa
- Department of Nursing, Faculty of Health and Welfare, Tokushima Bunri University, Tokushima 770-8514, Japan;
| | - Tetsuya Tanioka
- Institute of Biomedical Sciences, Tokushima University, Tokushima 770-8509, Japan; (T.T.); or (R.C.L.)
| | - Rozzano C. Locsin
- Institute of Biomedical Sciences, Tokushima University, Tokushima 770-8509, Japan; (T.T.); or (R.C.L.)
- Christine E Lynn College of Nursing, Florida Atlantic University, Boca Raton, FL 33431, USA
| |
Collapse
|
3
|
Betriana F, Tanioka R, Kogawa A, Suzuki R, Seki Y, Osaka K, Zhao Y, Kai Y, Tanioka T, Locsin R. Remote-Controlled Drone System through Eye Movements of Patients Who Need Long-Term Care: An Intermediary's Role. Healthcare (Basel) 2022; 10:827. [PMID: 35627964 PMCID: PMC9140421 DOI: 10.3390/healthcare10050827] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/24/2022] [Revised: 04/22/2022] [Accepted: 04/26/2022] [Indexed: 11/16/2022] Open
Abstract
The use of a remote-controlled drone system (RDS) by eye movements was studied to assist patients in psychiatric long-term care (PLTC) to allow them to view the environment outside the hospital, hoping that this will bring them some enjoyment. However, successfully applying this system requires human intermediaries in facilitating the interactions between patients and RDS operators. The aim of the study was to describe the role of nurses as intermediaries in the application of an RDS through eye movements of patients PLTC. This study employed the Intentional Observational Clinical Research Design. Data collection was performed in November 2021 at a psychiatric hospital with selected patients in PLTC. Seventeen patients took part in the indoor experiment, whereas 23 patients took part in the outdoor experiment. Fifteen of the 23 patients in the outdoor experiment were the same patients who took part in the indoor experiment. Most of the patients in the indoor and outdoor test arenas could successfully, delightfully, and safely fly the drone. This study demonstrated that RDS using just eye movements could increase the quality of life in older patients with psychiatric problems in PLTC. For the successful use of this drone system, nurse intermediaries assumed critically significant roles.
Collapse
Affiliation(s)
- Feni Betriana
- Graduate School of Health Sciences, Tokushima University, Tokushima 770-8503, Japan; (F.B.); (R.T.)
| | - Ryuichi Tanioka
- Graduate School of Health Sciences, Tokushima University, Tokushima 770-8503, Japan; (F.B.); (R.T.)
| | - Atsunori Kogawa
- Department of Mechanical Engineering, Tokai University, Kanagawa 259-1292, Japan; (A.K.); (R.S.); (Y.S.); (Y.K.)
| | - Riku Suzuki
- Department of Mechanical Engineering, Tokai University, Kanagawa 259-1292, Japan; (A.K.); (R.S.); (Y.S.); (Y.K.)
| | - Yuki Seki
- Department of Mechanical Engineering, Tokai University, Kanagawa 259-1292, Japan; (A.K.); (R.S.); (Y.S.); (Y.K.)
| | - Kyoko Osaka
- Department of Clinical Nursing, Kochi Medical School, Kochi University, Nankoku 783-8505, Japan
| | - Yueren Zhao
- Department of Psychiatry, Fujita Health University, Aichi 470-1192, Japan;
| | - Yoshihiro Kai
- Department of Mechanical Engineering, Tokai University, Kanagawa 259-1292, Japan; (A.K.); (R.S.); (Y.S.); (Y.K.)
| | - Tetsuya Tanioka
- Institute of Biomedical Sciences, Graduate School, Tokushima University, Tokushima 770-8509, Japan; (T.T.); (R.L.)
| | - Rozzano Locsin
- Institute of Biomedical Sciences, Graduate School, Tokushima University, Tokushima 770-8509, Japan; (T.T.); (R.L.)
| |
Collapse
|
4
|
Betriana F, Tanioka R, Yokotani T, Matsumoto K, Zhao Y, Osaka K, Miyagawa M, Kai Y, Schoenhofer S, Locsin RC, Tanioka T. Characteristics of interactive communication between Pepper robot, patients with schizophrenia, and healthy persons. BELITUNG NURSING JOURNAL 2022; 8:176-184. [PMID: 37521889 PMCID: PMC10386810 DOI: 10.33546/bnj.1998] [Citation(s) in RCA: 2] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/13/2021] [Revised: 01/17/2022] [Accepted: 03/12/2022] [Indexed: 08/01/2023] Open
Abstract
Background Expressing enjoyment when conversing with healthcare robots is an opportunity to enhance the value of human robots with interactive capabilities. In clinical practice, it is common to find verbal dysfunctions in patients with schizophrenia. Thus, interactive communication characteristics may vary between Pepper robot, persons with schizophrenia, and healthy persons. Objective Two case studies aimed to describe the characteristics of interactive communications, 1) between Pepper as a healthcare robot and two patients with schizophrenia, and 2) between Pepper as a healthcare robot and two healthy persons. Case Report The "Intentional Observational Clinical Research Design" was used to collect data. Using audio-video technology, the conversational interactions between the four participants with the Pepper healthcare robot were recorded. Their interactions were observed, with significant events noted. After their interactions, the four participants were interviewed regarding their experience and impressions of interacting with the Pepper healthcare robot. Audio-video recordings were analyzed following the analysis and interpretation protocol, and the interview data were transcribed, analyzed, and interpreted. Discussion There were similarities and differences in the interactive communication characteristics between the Pepper robot and the two participants with schizophrenia and between Pepper and the two healthy participants. The similarities were experiences of human enjoyment while interacting with the Pepper robot. This enjoyment was enhanced with the expectancy of the Pepper robot as able to entertain, and possessing interactive capabilities, indicating two-way conversational abilities. However, different communicating characteristics were found between the healthy participants' impressions of the Pepper robot and the participants with schizophrenia. Healthy participants understood Pepper to be an automaton, with responses to questions often constrained and, on many occasions, displaying inaccurate gaze. Conclusion Pepper robot showed capabilities for effective communication pertaining to expressing enjoyment. The accuracy and appropriateness of gaze remained a critical characteristic regardless of the situation or occasion with interactions between persons with schizophrenia, and between healthy persons. It is important to consider that in the future, for effective use of healthcare robots with multiple users, improvements in the areas of the appropriateness of gaze, response time during the conversation, and entertaining functions are critically observed.
Collapse
Affiliation(s)
- Feni Betriana
- Graduate School of Health Sciences, Tokushima University, Tokushima, Japan
| | - Ryuichi Tanioka
- Graduate School of Health Sciences, Tokushima University, Tokushima, Japan
| | - Tomoya Yokotani
- Graduate School of Health Sciences, Tokushima University, Tokushima, Japan
| | - Kazuyuki Matsumoto
- Graduate School of Technology, Industrial and Social Sciences, Tokushima University, Tokushima, Japan
| | - Yueren Zhao
- Department of Psychiatry, School of Medicine, Fujita Health University, Aichi, Japan
| | - Kyoko Osaka
- Department of Clinical Nursing, Kochi Medical School, Kochi University, Kochi, Japan
| | - Misao Miyagawa
- Department of Nursing, Faculty of Health and Welfare, Tokushima Bunri University, Tokushima, Japan
| | - Yoshihiro Kai
- Department of Mechanical Engineering, Tokai University, Kanagawa, Japan
| | - Savina Schoenhofer
- Anne Boykin Institute, Florida Atlantic University, Boca Raton, FL 33431–0991, USA
| | - Rozzano C. Locsin
- Tokushima University, Tokushima, Japan
- Florida Atlantic University, Boca Raton, FL 33431, USA
| | - Tetsuya Tanioka
- Department of Nursing Outcome Management, Institute of Biomedical Sciences, Tokushima University, Tokushima, Japan
| |
Collapse
|
5
|
|