1
|
Raza A, Rehman A, Sehar R, Alamri FS, Alotaibi S, Al Ghofaily B, Saba T. Optimized virtual reality design through user immersion level detection with novel feature fusion and explainable artificial intelligence. PeerJ Comput Sci 2024; 10:e2150. [PMID: 39145242 PMCID: PMC11323078 DOI: 10.7717/peerj-cs.2150] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/16/2024] [Accepted: 05/31/2024] [Indexed: 08/16/2024]
Abstract
Virtual reality (VR) and immersive technology have emerged as powerful tools with numerous applications. VR technology creates a computer-generated simulation that immerses users in a virtual environment, providing a highly realistic and interactive experience. This technology finds applications in various fields, including gaming, healthcare, education, architecture, and training simulations. Understanding user immersion levels in VR is crucial and challenging for optimizing the design of VR applications. Immersion refers to the extent to which users feel absorbed and engrossed in the virtual environment. This research primarily aims to detect user immersion levels in VR using an efficient machine-learning model. We utilized a benchmark dataset based on user experiences in VR environments to conduct our experiments. Advanced deep and machine learning approaches are applied in comparison. We proposed a novel technique called Polynomial Random Forest (PRF) for feature generation mechanisms. The proposed PRF approach extracts polynomial and class prediction probability features to generate a new feature set. Extensive research experiments show that random forest outperformed state-of-the-art approaches, achieving a high immersion level detection rate of 98%, using the proposed PRF technique. We applied hyperparameter optimization and cross-validation approaches to validate the performance scores. Additionally, we utilized explainable artificial intelligence (XAI) to interpret the reasoning behind the decisions made by the proposed model for user immersion level detection in VR. Our research has the potential to revolutionize user immersion level detection in VR, enhancing the design process.
Collapse
Affiliation(s)
- Ali Raza
- Department of Software Engineering, University of Lahore, Lahore, Pakistan
| | - Amjad Rehman
- Artificial Intelligence & Data Analytics Lab CCIS, Prince Sultan University, Riyadh, Saudi Arabia
| | - Rukhshanda Sehar
- Institute of Computer Science, Khwaja Fareed University of Engineering and Information Technology, Rahim Yar Khan, Pakistan
| | - Faten S. Alamri
- Department of Mathematical Sciences, College of Science, Princess Nourah Bint Abdulrahman University, Riyadh, Saudi Arabia
| | - Sarah Alotaibi
- Department of Computer Science, College of Computer and Information Sciences, King Saud University, Riyadh, Saudi Arabia
| | - Bayan Al Ghofaily
- Artificial Intelligence & Data Analytics Lab CCIS, Prince Sultan University, Riyadh, Saudi Arabia
| | - Tanzila Saba
- Artificial Intelligence & Data Analytics Lab CCIS, Prince Sultan University, Riyadh, Saudi Arabia
| |
Collapse
|
2
|
Kim C, Kim C, Kim H, Kwak H, Lee W, Im CH. Facial electromyogram-based facial gesture recognition for hands-free control of an AR/VR environment: optimal gesture set selection and validation of feasibility as an assistive technology. Biomed Eng Lett 2023; 13:465-473. [PMID: 37519877 PMCID: PMC10382369 DOI: 10.1007/s13534-023-00277-9] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/06/2023] [Revised: 04/01/2023] [Accepted: 04/03/2023] [Indexed: 08/01/2023] Open
Abstract
The rapid expansion of virtual reality (VR) and augmented reality (AR) into various applications has increased the demand for hands-free input interfaces when traditional control methods are inapplicable (e.g., for paralyzed individuals who cannot move their hands). Facial electromyogram (fEMG), bioelectric signals generated from facial muscles, could solve this problem. Discriminating facial gestures using fEMG is possible because fEMG signals vary with these gestures. Thus, these signals can be used to generate discrete hands-free control commands. This study implemented an fEMG-based facial gesture recognition system for generating discrete commands to control an AR or VR environment. The fEMG signals around the eyes were recorded, assuming that the fEMG electrodes were embedded into the VR head-mounted display (HMD). Sixteen discrete facial gestures were classified using linear discriminant analysis (LDA) with Riemannian geometry features. Because the fEMG electrodes were far from the facial muscles associated with the facial gestures, some similar facial gestures were indistinguishable from each other. Therefore, this study determined the best facial gesture combinations with the highest classification accuracy for 3-15 commands. An analysis of the fEMG data acquired from 15 participants showed that the optimal facial gesture combinations increased the accuracy by 4.7%p compared with randomly selected facial gesture combinations. Moreover, this study is the first to investigate the feasibility of implementing a subject-independent facial gesture recognition system that does not require individual user training sessions. Lastly, our online hands-free control system was successfully applied to a media player to demonstrate the applicability of the proposed system. Supplementary Information The online version contains supplementary material available at 10.1007/s13534-023-00277-9.
Collapse
Affiliation(s)
- Chunghwan Kim
- Department of Electronic Engineering, Hanyang University, Seoul, 04763 Republic of Korea
| | - Chaeyoon Kim
- Department of HY-KIST Bio-Convergence, Hanyang University, Seoul, 04763 Republic of Korea
| | - HyunSub Kim
- Department of Electronic Engineering, Hanyang University, Seoul, 04763 Republic of Korea
| | - HwyKuen Kwak
- Hanwha Systems Co., Ltd., Seongnam, 13524 Republic of Korea
| | - WooJin Lee
- Korea Research Institute for Defense Technology Planning and Advancement, Jinju, 52851 Republic of Korea
| | - Chang-Hwan Im
- Department of Electronic Engineering, Hanyang University, Seoul, 04763 Republic of Korea
- Department of HY-KIST Bio-Convergence, Hanyang University, Seoul, 04763 Republic of Korea
- Department of Biomedical Engineering, Hanyang University, Seoul, 04763 Republic of Korea
| |
Collapse
|
3
|
Kim C, Cha HS, Kim J, Kwak H, Lee W, Im CH. Facial Motion Capture System Based on Facial Electromyogram and Electrooculogram for Immersive Social Virtual Reality Applications. SENSORS (BASEL, SWITZERLAND) 2023; 23:3580. [PMID: 37050641 PMCID: PMC10099104 DOI: 10.3390/s23073580] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 01/12/2023] [Revised: 02/28/2023] [Accepted: 03/27/2023] [Indexed: 06/19/2023]
Abstract
With the rapid development of virtual reality (VR) technology and the market growth of social network services (SNS), VR-based SNS have been actively developed, in which 3D avatars interact with each other on behalf of the users. To provide the users with more immersive experiences in a metaverse, facial recognition technologies that can reproduce the user's facial gestures on their personal avatar are required. However, it is generally difficult to employ traditional camera-based facial tracking technology to recognize the facial expressions of VR users because a large portion of the user's face is occluded by a VR head-mounted display (HMD). To address this issue, attempts have been made to recognize users' facial expressions based on facial electromyogram (fEMG) recorded around the eyes. fEMG-based facial expression recognition (FER) technology requires only tiny electrodes that can be readily embedded in the HMD pad that is in contact with the user's facial skin. Additionally, electrodes recording fEMG signals can simultaneously acquire electrooculogram (EOG) signals, which can be used to track the user's eyeball movements and detect eye blinks. In this study, we implemented an fEMG- and EOG-based FER system using ten electrodes arranged around the eyes, assuming a commercial VR HMD device. Our FER system could continuously capture various facial motions, including five different lip motions and two different eyebrow motions, from fEMG signals. Unlike previous fEMG-based FER systems that simply classified discrete expressions, with the proposed FER system, natural facial expressions could be continuously projected on the 3D avatar face using machine-learning-based regression with a new concept named the virtual blend shape weight, making it unnecessary to simultaneously record fEMG and camera images for each user. An EOG-based eye tracking system was also implemented for the detection of eye blinks and eye gaze directions using the same electrodes. These two technologies were simultaneously employed to implement a real-time facial motion capture system, which could successfully replicate the user's facial expressions on a realistic avatar face in real time. To the best of our knowledge, the concurrent use of fEMG and EOG for facial motion capture has not been reported before.
Collapse
Affiliation(s)
- Chunghwan Kim
- Department of Electronic Engineering, Hanyang University, Seoul 04763, Republic of Korea; (C.K.); (H.-S.C.); (J.K.)
| | - Ho-Seung Cha
- Department of Electronic Engineering, Hanyang University, Seoul 04763, Republic of Korea; (C.K.); (H.-S.C.); (J.K.)
| | - Junghwan Kim
- Department of Electronic Engineering, Hanyang University, Seoul 04763, Republic of Korea; (C.K.); (H.-S.C.); (J.K.)
| | - HwyKuen Kwak
- Hanwha Systems Co., Ltd., Seongnam 13524, Republic of Korea;
| | - WooJin Lee
- Korea Research Institute for defense Technology Planning and Advancement, Jinju 52851, Republic of Korea;
| | - Chang-Hwan Im
- Department of Electronic Engineering, Hanyang University, Seoul 04763, Republic of Korea; (C.K.); (H.-S.C.); (J.K.)
- Department of Biomedical Engineering, Hanyang University, Seoul 04763, Republic of Korea
| |
Collapse
|
4
|
Expressive Interaction Design Using Facial Muscles as Controllers. MULTIMODAL TECHNOLOGIES AND INTERACTION 2022. [DOI: 10.3390/mti6090078] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/16/2022] Open
Abstract
Here we describe a proof-of-concept case study focusing on the design and development of a novel computer interface that uses facial muscles to control interactivity within a virtual environment. We have developed a system comprised of skin-mounted electrodes that detect underlying muscle activity through electromyography. The signals from the electrodes are filtered and smoothed, then used as input data to an application that displays a virtual environment with a 3D animated avatar. The user’s expressions control the facial movements of the avatar, thus conveying user emotions through real-time animation of a representative face in a virtual scenario. To achieve this, we collaborated with our Public and Patient Involvement focus group to discuss concepts and design appropriate interactions, while simultaneously developing a prototype system. Programmers and 3D artists worked together to create a system whereby individual user facial muscles are connected to 3D animated models of the same muscle features represented in an avatar, providing the user with an option to receive visual and numerical feedback on the extent of their muscle control. Using the prototype system, people can communicate facial expressions virtually with each other, without the need for a camera. This research is part of an on-going project to develop a facial muscle rehabilitation system that can be optimized to help patients with conditions such as hypomimia.
Collapse
|
5
|
Srinivasan B, Iqbal MU, Shahab MA, Srinivasan R. Review of Virtual Reality (VR) Applications To Enhance Chemical Safety: From Students to Plant Operators. ACS CHEMICAL HEALTH & SAFETY 2022. [DOI: 10.1021/acs.chas.2c00006] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/30/2022]
Affiliation(s)
- Babji Srinivasan
- Department of Applied Mechanics, Indian Institute of Technology Madras, Chennai 600036, India
- American Express Lab for Data Analytics, Risk & Technology, Indian Institute of Technology Madras, Chennai 600036, India
| | - Mohd Umair Iqbal
- Department of Chemical Engineering, Indian Institute of Technology Gandhinagar, Gandhinagar 382355, India
| | - Mohammed Aatif Shahab
- Department of Applied Mechanics, Indian Institute of Technology Madras, Chennai 600036, India
| | - Rajagopalan Srinivasan
- Department of Chemical Engineering, Indian Institute of Technology Madras, Chennai 600036, India
- American Express Lab for Data Analytics, Risk & Technology, Indian Institute of Technology Madras, Chennai 600036, India
| |
Collapse
|