Pupil Localisation and Eye Centre Estimation Using Machine Learning and Computer Vision

Eye Segmentation Method for Telehealth: Application to the Myasthenia Gravis Physical Examination

Due to the precautions put in place during the COVID-19 pandemic, utilization of telemedicine has increased quickly for patient care and clinical trials. Unfortunately, teleconsultation is closer to a video conference than a medical consultation, with the current solutions setting the patient and doctor into an evaluation that relies entirely on a two-dimensional view of each other. We are developing a patented telehealth platform that assists with diagnostic testing of ocular manifestations of myasthenia gravis. We present a hybrid algorithm combining deep learning with computer vision to give quantitative metrics of ptosis and ocular muscle fatigue leading to eyelid droop and diplopia. The method works both on a fixed image and frame by frame of the video in real-time, allowing capture of dynamic muscular weakness during the examination. We then use signal processing and filtering to derive robust metrics of ptosis and l ocular misalignment. In our construction, we have prioritized the robustness of the method versus accuracy obtained in controlled conditions in order to provide a method that can operate in standard telehealth conditions. The approach is general and can be applied to many disorders of ocular motility and ptosis.

The use of gaze to study cognition: limitations, solutions, and applications to animal welfare

The study of gaze responses, typically using looking time paradigms, has become a popular approach to improving our understanding of cognitive processes in non-verbal individuals. Our interpretation of data derived from these paradigms, however, is constrained by how we conceptually and methodologically approach these problems. In this perspective paper, we outline the application of gaze studies in comparative cognitive and behavioral research and highlight current limitations in the interpretation of commonly used paradigms. Further, we propose potential solutions, including improvements to current experimental approaches, as well as broad-scale benefits of technology and collaboration. Finally, we outline the potential benefits of studying gaze responses from an animal welfare perspective. We advocate the implementation of these proposals across the field of animal behavior and cognition to aid experimental validity, and further advance our knowledge on a variety of cognitive processes and welfare outcomes.

Vision-Based Eye Image Classification for Ophthalmic Measurement Systems

The accuracy and the overall performances of ophthalmic instrumentation, where specific analysis of eye images is involved, can be negatively influenced by invalid or incorrect frames acquired during everyday measurements of unaware or non-collaborative human patients and non-technical operators. Therefore, in this paper, we investigate and compare the adoption of several vision-based classification algorithms belonging to different fields, i.e., Machine Learning, Deep Learning, and Expert Systems, in order to improve the performance of an ophthalmic instrument designed for the Pupillary Light Reflex measurement. To test the implemented solutions, we collected and publicly released PopEYE as one of the first datasets consisting of 15 k eye images belonging to 22 different subjects acquired through the aforementioned specialized ophthalmic device. Finally, we discuss the experimental results in terms of classification accuracy of the eye status, as well as computational load analysis, since the proposed solution is designed to be implemented in embedded boards, which have limited hardware resources in computational power and memory size.

Motion-Based Object Location on a Smart Image Sensor Using On-Pixel Memory

Object location is a crucial computer vision method often used as a previous stage to object classification. Object-location algorithms require high computational and memory resources, which poses a difficult challenge for portable and low-power devices, even when the algorithm is implemented using dedicated digital hardware. Moving part of the computation to the imager may reduce the memory requirements of the digital post-processor and exploit the parallelism available in the algorithm. This paper presents the architecture of a Smart Imaging Sensor (SIS) that performs object location using pixel-level parallelism. The SIS is based on a custom smart pixel, capable of computing frame differences in the analog domain, and a digital coprocessor that performs morphological operations and connected components to determine the bounding boxes of the detected objects. The smart-pixel array implements on-pixel temporal difference computation using analog memories to detect motion between consecutive frames. Our SIS can operate in two modes: (1) as a conventional image sensor and (2) as a smart sensor which delivers a binary image that highlights the pixels in which movement is detected between consecutive frames and the object bounding boxes. In this paper, we present the design of the smart pixel and evaluate its performance using post-parasitic extraction on a 0.35 µm mixed-signal CMOS process. With a pixel-pitch of 32 µm × 32 µm, we achieved a fill factor of 28%. To evaluate the scalability of the design, we ported the layout to a 0.18 µm process, achieving a fill factor of 74%. On an array of 320×240 smart pixels, the circuit operates at a maximum frame rate of 3846 frames per second. The digital coprocessor was implemented and validated on a Xilinx Artix-7 XC7A35T field-programmable gate array that runs at 125 MHz, locates objects in a video frame in 0.614 µs, and has a power consumption of 58 mW.

Segmentation Method of Cerebral Aneurysms Based on Entropy Selection Strategy

The segmentation of cerebral aneurysms is a challenging task because of their similar imaging features to blood vessels and the great imbalance between the foreground and background. However, the existing 2D segmentation methods do not make full use of 3D information and ignore the influence of global features. In this study, we propose an automatic solution for the segmentation of cerebral aneurysms. The proposed method relies on the 2D U-Net as the backbone and adds a Transformer block to capture remote information. Additionally, through the new entropy selection strategy, the network pays more attention to the indistinguishable blood vessels and aneurysms, so as to reduce the influence of class imbalance. In order to introduce global features, three continuous patches are taken as inputs, and a segmentation map corresponding to the central patch is generated. In the inference phase, using the proposed recombination strategy, the segmentation map was generated, and we verified the proposed method on the CADA dataset. We achieved a Dice coefficient (DSC) of 0.944, an IOU score of 0.941, recall of 0.946, an F2 score of 0.942, a mAP of 0.896 and a Hausdorff distance of 3.12 mm.

Monocular Depth Estimation Using Deep Learning: A Review

In current decades, significant advancements in robotics engineering and autonomous vehicles have improved the requirement for precise depth measurements. Depth estimation (DE) is a traditional task in computer vision that can be appropriately predicted by applying numerous procedures. This task is vital in disparate applications such as augmented reality and target tracking. Conventional monocular DE (MDE) procedures are based on depth cues for depth prediction. Various deep learning techniques have demonstrated their potential applications in managing and supporting the traditional ill-posed problem. The principal purpose of this paper is to represent a state-of-the-art review of the current developments in MDE based on deep learning techniques. For this goal, this paper tries to highlight the critical points of the state-of-the-art works on MDE from disparate aspects. These aspects include input data shapes and training manners such as supervised, semi-supervised, and unsupervised learning approaches in combination with applying different datasets and evaluation indicators. At last, limitations regarding the accuracy of the DL-based MDE models, computational time requirements, real-time inference, transferability, input images shape and domain adaptation, and generalization are discussed to open new directions for future research.

Stateful-Service-Based Pupil Recognition in Natural Light Environments

Smartphones are currently extensively used worldwide, and advances in hardware quality have engendered improvements in smartphone image quality, which is occasionally comparable to the quality of medical imaging systems. This paper proposes two algorithms for pupil recognition: a stateful-service-based pupil recognition mechanism and color component low-pass filtering algorithm. The PRSSM algorithm can determine pupil diameters in images captured in indoor natural light environments, and the CCLPF algorithm can determine pupil diameters in those captured outdoors under sunlight. The PRSSM algorithm converts RGB colors into the hue saturation value color space and performs adaptive thresholding, morphological operations, and contour detection for effectively analyzing the diameter of the pupil. The CCLPF algorithm derives the average matrix for the red components of eye images captured in outdoor environments. It also performs low-pass filtering, morphological and contour detection operations, and rule-of-thumb correction. This algorithm can effectively analyze pupil diameter in outdoor natural light. Traditional ruler-based measurements of pupil diameter were used as the reference to verify the accuracy of the PRSSM and CCLPF algorithms and to compare their accuracy with that of the other algorithm. The errors in pupil diameter data were smaller for the PRSSM and CCLPF algorithms than for the other algorithm.

Automatic Asbestos Control Using Deep Learning Based Computer Vision System

The paper discusses the results of the research and development of an innovative deep learning-based computer vision system for the fully automatic asbestos content (productivity) estimation in rock chunk (stone) veins in an open pit and within the time comparable with the work of specialists (about 10 min per one open pit processing place). The discussed system is based on the applying of instance and semantic segmentation of artificial neural networks. The Mask R-CNN-based network architecture is applied to the asbestos-containing rock chunks searching images of an open pit. The U-Net-based network architecture is applied to the segmentation of asbestos veins in the images of selected rock chunks. The designed system allows an automatic search and takes images of the asbestos rocks in an open pit in the near-infrared range (NIR) and processes the obtained images. The result of the system work is the average asbestos content (productivity) estimation for each controlled open pit. It is validated to estimate asbestos content as the graduated average ratio of the vein area value to the selected rock chunk area value, both determined by the trained neural network. For both neural network training tasks the training, validation, and test datasets are collected. The designed system demonstrates an error of about 0.4% under different weather conditions in an open pit when the asbestos content is about 1.5–4%. The obtained accuracy is sufficient to use the system as a geological service tool instead of currently applied visual-based estimations.

The relationship between non-motor features and weight-loss in the premanifest stage of Huntington's disease

Weight-loss is an integral part of Huntington's disease (HD) that can start before the onset of motor symptoms. Investigating the underlying pathological processes may help in the understanding of this devastating disease as well as contribute to its management. However, the complex behavior and associations of multiple biological factors is impractical to be interpreted by the conventional statistics or human experts. For the first time, we combine a clinical dataset, expert knowledge and machine intelligence to model the multi-dimensional associations between the potentially relevant factors and weight-loss activity in HD, specifically at the premanifest stage. The HD dataset is standardized and transformed into required knowledge base with the help of clinical HD experts, which is then processed by the class rule mining and self-organising maps to identify the significant associations. Statistical results and experts' report indicate a strong association between severe weight-loss in HD at the premanifest stage and measures of certain cognitive, psychiatric functional ability factors. These results suggest that the mechanism underlying weight-loss in HD is, at least partly related to dysfunction of certain areas of the brain, a finding that may have not been apparent otherwise. These associations will aid the understanding of the pathophysiology of the disease and its progression and may in turn help in HD treatment trials.

Monocular Visual-Inertial Navigation for Dynamic Environment

Simultaneous localization and mapping (SLAM) systems have been generally limited to static environments. Moving objects considerably reduce the location accuracy of SLAM systems, rendering them unsuitable for several applications. Using a combined vision camera and inertial measurement unit (IMU) to separate moving and static objects in dynamic scenes, we improve the location accuracy and adaptability of SLAM systems in these scenes. We develop a moving object-matched feature points elimination algorithm that uses IMU data to eliminate matches on moving objects but retains them on stationary objects. Moreover, we develop a second algorithm to validate the IMU data to avoid erroneous data from influencing image feature points matching. We test the new algorithms with public datasets and in a real-world experiment. In terms of the root mean square error of the location absolute pose error, the proposed method exhibited higher positioning accuracy for the public datasets than the traditional algorithms. Compared with the closed-loop errors obtained by OKVIS-mono and VINS-mono, those obtained in the practical experiment were lower by 50.17% and 56.91%, respectively. Thus, the proposed method eliminates the matching points on moving objects effectively and achieves feature point matching results that are realistic.

Quantifying the separability of data classes in neural networks

We introduce the Generalized Discrimination Value (GDV) that measures, in a non-invasive manner, how well different data classes separate in each given layer of an artificial neural network. It turns out that, at the end of the training period, the GDV in each given layer L attains a highly reproducible value, irrespective of the initialization of the network's connection weights. In the case of multi-layer perceptrons trained with error backpropagation, we find that classification of highly complex data sets requires a temporal reduction of class separability, marked by a characteristic 'energy barrier' in the initial part of the GDV(L) curve. Even more surprisingly, for a given data set, the GDV(L) is running through a fixed 'master curve', independently from the total number of network layers. Finally, due to its invariance with respect to dimensionality, the GDV may serve as a useful tool to compare the internal representational dynamics of artificial neural networks with different architectures for neural architecture search or network compression; or even with brain activity in order to decide between different candidate models of brain function.

A Fast and Effective System for Analysis of Optokinetic Waveforms with a Low-Cost Eye Tracking Device

Optokinetic nystagmus (OKN) is an involuntary eye movement induced by motion of a large proportion of the visual field. It consists of a "slow phase (SP)" with eye movements in the same direction as the movement of the pattern and a "fast phase (FP)" with saccadic eye movements in the opposite direction. Study of OKN can reveal valuable information in ophthalmology, neurology and psychology. However, the current commercially available high-resolution and research-grade eye tracker is usually expensive. Methods & Results: We developed a novel fast and effective system combined with a low-cost eye tracking device to accurately quantitatively measure OKN eye movement. Conclusions: The experimental results indicate that the proposed method achieves fast and promising results in comparisons with several traditional approaches.

Experimental Verification of Objective Visual Fatigue Measurement Based on Accurate Pupil Detection of Infrared Eye Image and Multi-Feature Analysis

As the use of electronic displays increases rapidly, visual fatigue problems are also increasing. The subjective evaluation methods used for visual fatigue measurement have individual difference problems, while objective methods based on bio-signal measurement have problems regarding motion artifacts. Conventional eye image analysis-based visual fatigue measurement methods do not accurately characterize the complex changes in the appearance of the eye. To solve this problem, in this paper, an objective visual fatigue measurement method based on infrared eye image analysis is proposed. For accurate pupil detection, a convolutional neural network-based semantic segmentation method was used. Three features are calculated based on the pupil detection results: (1) pupil accommodation speed, (2) blink frequency, and (3) eye-closed duration. In order to verify the calculated features, differences in fatigue caused by changes in content color components such as gamma, color temperature, and brightness were compared with a reference video. The pupil detection accuracy was confirmed to be 96.63% based on the mean intersection over union. In addition, it was confirmed that all three features showed significant differences from the reference group; thus, it was verified that the proposed analysis method can be used for the objective measurement of visual fatigue.