Earable TEMPO: A Novel, Hands-Free Input Device that Uses the Movement of the Tongue Measured with a Wearable Ear Sensor

Effects of Food Properties and Aging on External Auditory Canal Movements Associated With Mastication

Few objective evaluations of external auditory canal movement during mastication have been conducted. This study investigated the extent to which age and physical properties influence such movement. The effects of food properties and aging on ear canal movement during mastication were investigated using an earable reliable chewing-count measurement device. We used such a device to study the effects of food properties and aging on ear canal movement associated with mastication. A main effect of the difference in hardness between the foods (F = 8.3405, p = 0.0071) was found. No interaction (F = 1.3558, p = 0.2534) or main effect of age (F = 1.1206, p = 0.2982) was found. The values for peanuts were higher than those for pudding. Age had no significant effect. For both pudding and peanuts, there was a trend toward greater ear canal movement in young adults than in older adults. We suggest that external auditory canal movement during mastication decreases as muscle function declines with aging, but any effect may be less than that exerted by food properties.

Health-Related Indicators Measured Using Earable Devices: Systematic Review

BACKGROUND

Earable devices are novel, wearable Internet of Things devices that are user-friendly and have potential applications in mobile health care. The position of the ear is advantageous for assessing vital status and detecting diseases through reliable and comfortable sensing devices.

OBJECTIVE

Our study aimed to review the utility of health-related indicators derived from earable devices and propose an improved definition of disease prevention. We also proposed future directions for research on the health care applications of earable devices.

METHODS

A systematic review was conducted of the PubMed, Embase, and Web of Science databases. Keywords were used to identify studies on earable devices published between 2015 and 2020. The earable devices were described in terms of target health outcomes, biomarkers, sensor types and positions, and their utility for disease prevention.

RESULTS

A total of 51 articles met the inclusion criteria and were reviewed, and the frequency of 5 health-related characteristics of earable devices was described. The most frequent target health outcomes were diet-related outcomes (9/51, 18%), brain status (7/51, 14%), and cardiovascular disease (CVD) and central nervous system disease (5/51, 10% each). The most frequent biomarkers were electroencephalography (11/51, 22%), body movements (6/51, 12%), and body temperature (5/51, 10%). As for sensor types and sensor positions, electrical sensors (19/51, 37%) and the ear canal (26/51, 51%) were the most common, respectively. Moreover, the most frequent prevention stages were secondary prevention (35/51, 69%), primary prevention (12/51, 24%), and tertiary prevention (4/51, 8%). Combinations of ≥2 target health outcomes were the most frequent in secondary prevention (8/35, 23%) followed by brain status and CVD (5/35, 14% each) and by central nervous system disease and head injury (4/35, 11% each).

CONCLUSIONS

Earable devices can provide biomarkers for various health outcomes. Brain status, healthy diet status, and CVDs were the most frequently targeted outcomes among the studies. Earable devices were mostly used for secondary prevention via monitoring of health or disease status. The potential utility of earable devices for primary and tertiary prevention needs to be investigated further. Earable devices connected to smartphones or tablets through cloud servers will guarantee user access to personal health information and facilitate comfortable wearing.

Earable Ω (OMEGA): A Novel Clenching Interface Using Ear Canal Sensing for Human Metacarpophalangeal Joint Control by Functional Electrical Stimulation

(1) Background: A mouth-free interface is required for functional electrical stimulation (FES) in people with spinal cord injuries. We developed a novel system for clenching the human metacarpophalangeal (MP) joint using an earphone-type ear canal movement sensor. Experiments to control joint angle and joint stiffness were performed using the developed system. (2) Methods: The proposed FES used an equilibrium point control signal and stiffness control signal: electrical agonist-antagonist ratio and electrical agonist-antagonist sum. An angle sensor was used to acquire the joint angle, and system identification was utilized to measure joint stiffness using the external force of a robot arm. Each experiment included six and five subjects, respectively. (3) Results: While the joint angle could be controlled well by clenching with some hysteresis and delay in three subjects, it could not be controlled relatively well after hyperextension in the other subjects, which revealed a calibration problem and a change in the characteristics of the human MP joint caused by hyperextension. The joint stiffness increased with the clenching amplitude in five subjects. In addition, the results indicated that viscosity can be controlled. (4) Conclusions: The developed system can control joint angle and stiffness. In future research, we will develop a method to show that this system can control the equilibrium point and stiffness simultaneously.

Validation of Earphone-Type Sensors for Non-Invasive and Objective Swallowing Function Assessment

Standard methods for swallowing function evaluation are videofluoroscopy (VF) and videoendoscopy, which are invasive and have test limitations. We examined the use of an earphone-type sensor to noninvasively evaluate soft palate movement in comparison with VF. Six healthy adults wore earphone sensors and swallowed barium water while being filmed by VF. A light-emitting diode at the sensor tip irradiated infrared light into the ear canal, and a phototransistor received the reflected light to detect changes in ear canal movement, including that of the eardrum. Considering that the soft palate movement corresponded to the sensor waveform, a Bland–Altman analysis was performed on the difference in time recorded by each measurement method. The average difference between the time taken from the most downward retracted position before swallowing to the most upward position during swallowing of the soft palate in VF was −0.01 ± 0.14 s. The Bland–Altman analysis showed no fixed or proportional error. The minimal detectable change was 0.28 s. This is the first noninvasive swallowing function evaluation through the ear canal. The earphone-type sensor enabled us to measure the time from the most retracted to the most raised soft palate position during swallowing and validated this method for clinical application.

Auritus: An Open-Source Optimization Toolkit for Training and Development of Human Movement Models and Filters Using Earables

Smart ear-worn devices (called earables) are being equipped with various onboard sensors and algorithms, transforming earphones from simple audio transducers to multi-modal interfaces making rich inferences about human motion and vital signals. However, developing sensory applications using earables is currently quite cumbersome with several barriers in the way. First, time-series data from earable sensors incorporate information about physical phenomena in complex settings, requiring machine-learning (ML) models learned from large-scale labeled data. This is challenging in the context of earables because large-scale open-source datasets are missing. Secondly, the small size and compute constraints of earable devices make on-device integration of many existing algorithms for tasks such as human activity and head-pose estimation difficult. To address these challenges, we introduce Auritus an extendable and open-source optimization toolkit designed to enhance and replicate earable applications. Auritus serves two primary functions. Firstly, Auritus handles data collection, pre-processing, and labeling tasks for creating customized earable datasets using graphical tools. The system includes an open-source dataset with 2.43 million inertial samples related to head and full-body movements, consisting of 34 head poses and 9 activities from 45 volunteers. Secondly, Auritus provides a tightly-integrated hardware-in-the-loop (HIL) optimizer and TinyML interface to develop lightweight and real-time machine-learning (ML) models for activity detection and filters for head-pose tracking. To validate the utlity of Auritus, we showcase three sample applications, namely fall detection, spatial audio rendering, and augmented reality (AR) interfacing. Auritus recognizes activities with 91% leave 1-out test accuracy (98% test accuracy) using real-time models as small as 6-13 kB. Our models are 98-740× smaller and 3-6% more accurate over the state-of-the-art. We also estimate head pose with absolute errors as low as 5 degrees using 20kB filters, achieving up to 1.6× precision improvement over existing techniques. We make the entire system open-source so that researchers and developers can contribute to any layer of the system or rapidly prototype their applications using our dataset and algorithms.

Exploring Gaze Movement Gesture Recognition Method for Eye-Based Interaction Using Eyewear with Infrared Distance Sensor Array

With the spread of eyewear devices, people are increasingly using information devices in various everyday situations. In these situations, it is important for eyewear devices to have eye-based interaction functions for simple hands-free input at a low cost. This paper proposes a gaze movement recognition method for simple hands-free interaction that uses eyewear equipped with an infrared distance sensor. The proposed method measures eyelid skin movement using an infrared distance sensor inside the eyewear and applies machine learning to the time-series sensor data to recognize gaze movements (e.g., up, down, left, and right). We implemented a prototype system and conducted evaluations with gaze movements including factors such as movement directions at 45-degree intervals and the movement distance difference in the same direction. The results showed the feasibility of the proposed method. The proposed method recognized 5 to 20 types of gaze movements with an F-value of 0.96 to 1.0. In addition, the proposed method was available with a limited number of sensors, such as two or three, and robust against disturbance in some usage conditions (e.g., body vibration, facial expression change). This paper provides helpful findings for the design of gaze movement recognition methods for simple hands-free interaction using eyewear devices at a low cost.

Augmenting Ear Accessories for Facial Gesture Input Using Infrared Distance Sensor Array

Simple hands-free input methods using ear accessories have been proposed to broaden the range of scenarios in which information devices can be operated without hands. Although many previous studies use canal-type earphones, few studies focused on the following two points: (1) A method applicable to ear accessories other than canal-type earphones. (2) A method enabling various ear accessories with different styles to have the same hands-free input function. To realize these two points, this study proposes a method to recognize the user’s facial gesture using an infrared distance sensor attached to the ear accessory. The proposed method detects skin movement around the ear and face, which differs for each facial expression gesture. We created a prototype system for three ear accessories for the root of the ear, earlobe, and tragus. The evaluation results for nine gestures and 10 subjects showed that the F-value of each device was 0.95 or more, and the F-value of the pattern combining multiple devices was 0.99 or more, which showed the feasibility of the proposed method. Although many ear accessories could not interact with information devices, our findings enable various ear accessories with different styles to have eye-free and hands-free input ability based on facial gestures.

Simultaneous Measurement of Ear Canal Movement, Electromyography of the Masseter Muscle and Occlusal Force for Earphone-Type Occlusal Force Estimation Device Development

We intend to develop earphone-type wearable devices to measure occlusal force by measuring ear canal movement using an ear sensor that we developed. The proposed device can measure occlusal force during eating. In this work, we simultaneously measured the ear canal movement (ear sensor value), the surface electromyography (EMG) of the masseter muscle and the occlusal force six times from five subjects as a basic study toward occlusal force meter development. Using the results, we investigated the correlation coefficient between the ear sensor value and the occlusal force, and the partial correlation coefficient between ear sensor values. Additionally, we investigated the average of the partial correlation coefficient and the absolute value of the average for each subject. The absolute value results indicated strong correlation, with correlation coefficients exceeding 0.9514 for all subjects. The subjects showed a lowest partial correlation coefficient of 0.6161 and a highest value of 0.8286. This was also indicative of correlation. We then estimated the occlusal force via a single regression analysis for each subject. Evaluation of the proposed method via the cross-validation method indicated that the root-mean-square error when comparing actual values with estimates for the five subjects ranged from 0.0338 to 0.0969.

Earable ZEN: Development of an Earphone-Type Zazen Support Wearable System

Meditation has been included in mental health care and the treatment of hypertension and pain. Zazen is a method of entering the meditative state. We have carried out R&D on a wearable system (earable ZEN) that casually and simply allows the subject to perform zazen without going to a temple or even having a zazen monitor present. In zazen, it is important to prepare one's breathing and posture. The earable ZEN is comprised of an earphone-type sensor (ear sensor) for measuring the breathing and posture of the user, a miniature actuator (neck belt) for communicating disturbances in the breathing and posture of the user, and a microcomputer. In an evaluation experiment, disturbance in breathing was simulated as “deep mouth breathing” and disturbance in posture was simulated as “nodding”. The average accuracy value for seven healthy subjects wearing the earable ZEN was 99.9% for mouth breathing and 100% for nodding. In the same way, the average precision value was 98.7% for mouth breathing and 100% for nodding, with an average recall value of 97.1% for mouth breathing and 100% for nodding. None of the subjects considered the ear sensor and neck belt to be an obstacle to zazen.

Earable POCER: Development of a Point-of-Care Ear Sensor for Respiratory Rate Measurement

We have carried out research and development on an earphone-type respiratory rate measuring device, earable POCER. The name earable POCER is a combination of "earable", which is a word coined from "wearable" and "ear", and "POCER", which is an acronym for "point-of-care ear sensor for respiratory rate measurement". The earable POCER calculates respiratory frequency, based on the measurement values over one minute, through the simple attachment of an ear sensor to one ear of the measured subject and displays these on a tablet terminal. The earable POCER irradiates infrared light using a light-emitting diode (LED) loaded on an ear sensor to the epidermis within the ear canal and, by receiving that reflected light with a phototransistor, it measures movement of the ear canal based on respiration. In an evaluation experiment, eight healthy subjects first breathed through the nose 12 times per minute, then 16 times per minute, and finally 20 times per minute, in accordance with the flashing of a timing instruction LED. The results of these evaluation tests showed that the accuracy of the respiratory frequency was 100% for nose breathing 12 times per minute, 93.8% at 16 times, and 93.8% at 20 times.

Mouthwitch: A Novel Head Mount Type Hands-Free Input Device that Uses the Movement of the Temple to Control a Camera

We have developed an interface (mouthwitch) for a head-mounted type camera with which pictures can be taken with a head-mounted camera, hands-free, simply by "opening your mouth continuously for approximately one second and then closing it again". This mouthwitch uses a sensor equipped with an LED and photo transistor on the temple to optically measure the changes in the form of the temple that occur when the mouth is opened and closed. Eight test subjects (males and females aged between 21 and 44 years old) performed evaluation tests using this mouthwitch when resting, speaking, chewing, walking, and running. The results showed that all test subjects were able to open and close the mouth, and the measurement results pertaining to the temple shape changes that occurred at this time were highly reproducible. Additionally, the average value for accuracy obtained for the eight test subjects through the verification tests was 100% when resting, chewing, or walking, and 99.8% when speaking or running. Similarly, the average values for precision were 100% for all items, and the average values for recall were 100% when resting or chewing, 98.8% when speaking, 97.5% when walking, and 87.5% when running.