Monitoring infants by automatic video processing: A unified approach to motion analysis

Computer vision for automated seizure detection and classification: A systematic review

Computer vision (CV) shows increasing promise as an efficient, low-cost tool for video seizure detection and classification. Here, we provide an overview of the fundamental concepts needed to understand CV and summarize the structure and performance of various model architectures used in video seizure analysis. We conduct a systematic literature review of the PubMed, Embase, and Web of Science databases from January 1, 2000 to September 15, 2023, to identify the strengths and limitations of CV seizure analysis methods and discuss the utility of these models when applied to different clinical seizure phenotypes. Reviews, nonhuman studies, and those with insufficient or poor quality data are excluded from the review. Of the 1942 records identified, 45 meet inclusion criteria and are analyzed. We conclude that the field has shown tremendous growth over the past 2 decades, leading to several model architectures with impressive accuracy and efficiency. The rapid and scalable detection offered by CV models holds the potential to reduce sudden unexpected death in epilepsy and help alleviate resource limitations in epilepsy monitoring units. However, a lack of standardized, thorough validation measures and concerns about patient privacy remain important obstacles for widespread acceptance and adoption. Investigation into the performance of models across varied datasets from clinical and nonclinical environments is an essential area for further research.

Automated analysis and detection of epileptic seizures in video recordings using artificial intelligence

Introduction

Automated seizure detection promises to aid in the prevention of SUDEP and improve the quality of care by assisting in epilepsy diagnosis and treatment adjustment.

Methods

In this phase 2 exploratory study, the performance of a contactless, marker-free, video-based motor seizure detection system is assessed, considering video recordings of patients (age 0-80 years), in terms of sensitivity, specificity, and Receiver Operating Characteristic (ROC) curves, with respect to video-electroencephalographic monitoring (VEM) as the medical gold standard. Detection performances of five categories of motor epileptic seizures (tonic-clonic, hyperkinetic, tonic, unclassified motor, automatisms) and psychogenic non-epileptic seizures (PNES) with a motor behavioral component lasting for >10 s were assessed independently at different detection thresholds (rather than as a categorical classification problem). A total of 230 patients were recruited in the study, of which 334 in-scope (>10 s) motor seizures (out of 1,114 total seizures) were identified by VEM reported from 81 patients. We analyzed both daytime and nocturnal recordings. The control threshold was evaluated at a range of values to compare the sensitivity (n = 81 subjects with seizures) and false detection rate (FDR) (n = all 230 subjects).

Results

At optimal thresholds, the performance of seizure groups in terms of sensitivity (CI) and FDR/h (CI): tonic-clonic- 95.2% (82.4, 100%); 0.09 (0.077, 0.103), hyperkinetic- 92.9% (68.5, 98.7%); 0.64 (0.59, 0.69), tonic- 78.3% (64.4, 87.7%); 5.87 (5.51, 6.23), automatism- 86.7% (73.5, 97.7%); 3.34 (3.12, 3.58), unclassified motor seizures- 78% (65.4, 90.4%); 4.81 (4.50, 5.14), and PNES- 97.7% (97.7, 100%); 1.73 (1.61, 1.86). A generic threshold recommended for all motor seizures under study asserted 88% sensitivity and 6.48 FDR/h.

Discussion

These results indicate an achievable performance for major motor seizure detection that is clinically applicable for use as a seizure screening solution in diagnostic workflows.

Multimodality Video Acquisition System for the Assessment of Vital Distress in Children

In children, vital distress events, particularly respiratory, go unrecognized. To develop a standard model for automated assessment of vital distress in children, we aimed to construct a prospective high-quality video database for critically ill children in a pediatric intensive care unit (PICU) setting. The videos were acquired automatically through a secure web application with an application programming interface (API). The purpose of this article is to describe the data acquisition process from each PICU room to the research electronic database. Using an Azure Kinect DK and a Flir Lepton 3.5 LWIR attached to a Jetson Xavier NX board and the network architecture of our PICU, we have implemented an ongoing high-fidelity prospectively collected video database for research, monitoring, and diagnostic purposes. This infrastructure offers the opportunity to develop algorithms (including computational models) to quantify vital distress in order to evaluate vital distress events. More than 290 RGB, thermographic, and point cloud videos of each 30 s have been recorded in the database. Each recording is linked to the patient's numerical phenotype, i.e., the electronic medical health record and high-resolution medical database of our research center. The ultimate goal is to develop and validate algorithms to detect vital distress in real time, both for inpatient care and outpatient management.

An Evaluation of Non-Contact Photoplethysmography-Based Methods for Remote Respiratory Rate Estimation

The respiration rate (RR) is one of the physiological signals deserving monitoring for assessing human health and emotional states. However, traditional devices, such as the respiration belt to be worn around the chest, are not always a feasible solution (e.g., telemedicine, device discomfort). Recently, novel approaches have been proposed aiming at estimating RR in a less invasive yet reliable way, requiring the acquisition and processing of contact or remote Photoplethysmography (contact reference and remote-PPG, respectively). The aim of this paper is to address the lack of systematic evaluation of proposed methods on publicly available datasets, which currently impedes a fair comparison among them. In particular, we evaluate two prominent families of PPG processing methods estimating Respiratory Induced Variations (RIVs): the first encompasses methods based on the direct extraction of morphological features concerning the RR; and the second group includes methods modeling respiratory artifacts adopting, in the most promising cases, single-channel blind source separation. Extensive experiments have been carried out on the public BP4D+ dataset, showing that the morphological estimation of RIVs is more reliable than those produced by a single-channel blind source separation method (both in contact and remote testing phases), as well as in comparison with a representative state-of-the-art Deep Learning-based approach for remote respiratory information estimation.

Methods for Detecting Abnormal Ventilation in Children - the Case Study of 13-Years old Pitt-Hopkins Girl

We present contactless technology measuring abnormal ventilation and compare it with polysomnography (PSG). A 13-years old girl with Pitt-Hopkins syndrome presented hyperpnoea periods with apneic spells. The PSG was conducted simultaneously with Emfit movement sensor (Emfit, Finland) and video camera with depth sensor (NEL, Finland). The respiratory efforts from PSG, Emfit sensor, and NEL were compared. In addition, we measured daytime breathing with tracheal microphone (PneaVox,France). The aim was to deepen the knowledge of daytime hyperpnoea periods and ensure that no upper airway obstruction was present during sleep. The signs of upper airway obstruction were not detected despite of minor sleep time. Monitoring respiratory effort with PSG is demanding in all patient groups. The used unobtrusive methods were capable to reveal breathing frequency and hyperpnoea periods. Every day diagnostics need technology like this for monitoring vital signs at hospital wards and at home from subjects with disabilities and co-operation difficulties.

Heart Rate Variability Analysis for Seizure Detection in Neonatal Intensive Care Units

In Neonatal Intensive Care Units (NICUs), the early detection of neonatal seizures is of utmost importance for a timely clinical intervention. Over the years, several neonatal seizure detection systems were proposed to detect neonatal seizures automatically and speed up seizure diagnosis, most based on the EEG signal analysis. Recently, research has focused on other possible seizure markers, such as electrocardiography (ECG). This work proposes an ECG-based NSD system to investigate the usefulness of heart rate variability (HRV) analysis to detect neonatal seizures in the NICUs. HRV analysis is performed considering time-domain, frequency-domain, entropy and multiscale entropy features. The performance is evaluated on a dataset of ECG signals from 51 full-term babies, 29 seizure-free. The proposed system gives results comparable to those reported in the literature: Area Under the Receiver Operating Characteristic Curve = 62%, Sensitivity = 47%, Specificity = 67%. Moreover, the system’s performance is evaluated in a real clinical environment, inevitably affected by several artefacts. To the best of our knowledge, our study proposes for the first time a multi-feature ECG-based NSD system that also offers a comparative analysis between babies suffering from seizures and seizure-free ones.

Video-Based Automatic Baby Motion Analysis for Early Neurological Disorder Diagnosis: State of the Art and Future Directions

Neurodevelopmental disorders (NDD) are impairments of the growth and development of the brain and/or central nervous system. In the light of clinical findings on early diagnosis of NDD and prompted by recent advances in hardware and software technologies, several researchers tried to introduce automatic systems to analyse the baby's movement, even in cribs. Traditional technologies for automatic baby motion analysis leverage contact sensors. Alternatively, remotely acquired video data (e.g., RGB or depth) can be used, with or without active/passive markers positioned on the body. Markerless approaches are easier to set up and maintain (without any human intervention) and they work well on non-collaborative users, making them the most suitable technologies for clinical applications involving children. On the other hand, they require complex computational strategies for extracting knowledge from data, and then, they strongly depend on advances in computer vision and machine learning, which are among the most expanding areas of research. As a consequence, also markerless video-based analysis of movements in children for NDD has been rapidly expanding but, to the best of our knowledge, there is not yet a survey paper providing a broad overview of how recent scientific developments impacted it. This paper tries to fill this gap and it lists specifically designed data acquisition tools and publicly available datasets as well. Besides, it gives a glimpse of the most promising techniques in computer vision, machine learning and pattern recognition which could be profitably exploited for children motion analysis in videos.

Bionic for Training: Smart Framework Design for Multisensor Mechatronic Platform Validation

Home monitoring supports the continuous improvement of the therapy by sharing data with healthcare professionals. It is required when life-threatening events can still occur after hospital discharge such as neonatal apnea. However, multiple sources of external noise could affect data quality and/or increase the misdetection rate. In this study, we developed a mechatronic platform for sensor characterizations and a framework to manage data in the context of neonatal apnea. The platform can simulate the movement of the abdomen in different plausible newborn positions by merging data acquired simultaneously from three-axis accelerometers and infrared sensors. We simulated nine apnea conditions combining three different linear displacements and body postures in the presence of self-generated external noise, showing how it is possible to reduce errors near to zero in phenomena detection. Finally, the development of a smart 8Ws-based software and a customizable mobile application were proposed to facilitate data management and interpretation, classifying the alerts to guarantee the correct information sharing without specialized skills.

RGB-D scene analysis in the NICU

Continuity of care is achieved in the neonatal intensive care unit (NICU) through careful documentation of all events of clinical significance, including clinical interventions and routine care events (e.g., feeding, diaper change, weighing, etc.). As a step towards automating this documentation process, we propose a scene recognition algorithm that can automatically identify key features in a single image of the patient environment, paired with a rule-based sentence generator to caption the scene. Color and depth video were obtained from 29 newborn patients from the Children's Hospital of Eastern Ontario (CHEO) using an Intel RealSense SR300 RGB-D camera and manual bedside event annotation. Image processing techniques are implemented to classify two lighting conditions: brightness level and phototherapy. A deep neural network is developed for three image classification tasks: on-going intervention, bed occupancy, and patient coverage. Transfer learning is leveraged in the feature extraction layers, such that weights learned from a generic data-rich task are applied to the clinical domain where data collection is complex and costly. Different depth fusion techniques are implemented and compared among classification tasks, where the depth and color data are fused as an RGB-D image (image fusion) or separately at various layers in the network (network fusion). Promising results were obtained with >84% sensitivity and >73% F1 measure across all context variables despite the large class imbalance. RGBD-based models are shown to outperform RGB models on most tasks. In general, a 4-channel image fusion and network fusion at the 11th layer of the VGG-16 architecture were preferred. Ultimately, achieving complete scene understanding through multimodal computer vision could form the basis for a semi-automated charting system to assist clinical staff.

Predicting apneic events in preterm infants using cardio-respiratory and movement features

BACKGROUND AND OBJECTIVE

Preterm neonates are prone to episodes of apnea, bradycardia and hypoxia (ABH) that can lead to neurological morbidities or even death. There is broad interest in developing methods for real-time prediction of ABH events to inform interventions that prevent or reduce their incidence and severity. Using advances in machine learning methods, this study develops an algorithm to predict ABH events.

METHODS

Following previous studies showing that respiratory instabilities are closely associated with bouts of movement, we present a modeling framework that can predict ABH events using both movement and cardio-respiratory features derived from routine clinical recordings. In 10 preterm infants, movement onsets and durations were estimated with a wavelet-based algorithm that quantified artifactual distortions of the photoplethysmogram signal. For prediction, cardio-respiratory features were created from time-delayed correlations of inter-beat and inter-breath intervals with past values; movement features were derived from time-delayed correlations with inter-breath intervals. Gaussian Mixture Models and Logistic Regression were used to develop predictive models of apneic events. Performance of the models was evaluated with ROC curves.

RESULTS

Performance of the prediction framework (mean AUC) was 0.77 ± 0.04 for 66 ABH events on training data from 7 infants. When grouped by the severity of the associated bradycardia during the ABH event, the framework was able to predict 83% and 75% of the most severe episodes in the 7-infant training set and 3-infant test set, respectively. Notably, inclusion of movement features significantly improved the predictions compared with modeling with only cardio-respiratory signals.

CONCLUSIONS

Our findings suggest that recordings of movement provide important information for predicting ABH events in preterm infants, and can inform preemptive interventions designed to reduce the incidence and severity of ABH events.

Video-based neonatal state assessment method for timing of procedures

BACKGROUND

Procedures should be performed when an infant is most receptive to disruptions in order to reduce the stress on the infant. However, frequent direct observations place a heavy burden on medical staff. There is therefore a need for a method for quantitatively and automatically evaluating the neonatal state.

METHODS

Ten infants in our hospital were enrolled in this study. The states of the infants were assessed by medical staff using the Brazelton Neonatal Behavioral Assessment Scale and were recorded on video at the same time. The recorded states were reclassified as activity levels, a new state classification method that includes middle activity, which is the appropriate time for a procedure. Using image analysis, motions of the infant were quantified as two indices: activity and pause time. Activity and pause time were compared for each activity level. The cutoff values of the indices were calculated, and the sensitivity and specificity of the middle activity were calculated.

RESULTS

There was a significant difference between all groups of activity level (P < 0.01). The maximum sensitivity and specificity of middle activity were 71.7% and 51.2%, respectively.

CONCLUSIONS

The neonatal state of infants can be quantitatively and automatically evaluated using video cameras, and the activity level can be used to determine an appropriate time for procedures in infants. This will reduce the burden on medical staff and lead to less stressful procedures for infants.

Detection of Breathing Movements of Preterm Neonates by Recording Their Abdominal Movements with a Time-of-Flight Camera

In order to deliver an aerosolized drug in a breath-triggered manner, the initiation of the patient’s inspiration needs to be detected. The best-known systems monitoring breathing patterns are based on flow sensors. However, due to their large dead space volume, flow sensors are not advisable for monitoring the breathing of (preterm) neonates. Newly-developed respiratory sensors, especially when contact-based (invasive), can be tested on (preterm) neonates only with great effort due to clinical and ethical hurdles. Therefore, a physiological model is highly desirable to validate these sensors. For developing such a system, abdominal movement data of (preterm) neonates are required. We recorded time sequences of five preterm neonates’ abdominal movements with a time-of-flight camera and successfully extracted various breathing patterns and respiratory parameters. Several characteristic breathing patterns, such as forced breathing, sighing, apnea and crying, were identified from the movement data. Respiratory parameters, such as duration of inspiration and expiration, as well as respiratory rate and breathing movement over time, were also extracted. This work demonstrated that respiratory parameters of preterm neonates can be determined without contact. Therefore, such a system can be used for breathing detection to provide a trigger signal for breath-triggered drug release systems. Furthermore, based on the recorded data, a physiological abdominal movement model of preterm neonates can now be developed.

Voxyvi: A system for long-term audio and video acquisitions in neonatal intensive care units

BACKGROUND

In the European Union, 300,000 newborn babies are born prematurely every year. Their care is ensured in Neonatal Intensive Care Units (NICU) where vital signs are constantly monitored. In addition, other descriptors such as motion, facial and vocal activities have been shown to be essential to assess neurobehavioral development.

AIM

In the scope of the European project Digi-NewB, we aimed to develop and evaluate a new audio-video device designed to non-invasively acquire multi-modal data (audio, video and thermal images), while fitting the wide variety of bedding environment in NICU.

METHODS

Firstly, a multimodal system and associated software and guidelines to collect data in neonatal intensive care unit were proposed. Secondly, methods for post-evaluation of the acquisition phase were developed, including the study of clinician feedback and a qualitative analysis of the data.

RESULTS

The deployment of 19 acquisition devices in six French hospitals allowed to record more than 500 newborns of different gestational and postmenstrual ages. After the acquisition phase, clinical feedback was mostly positive. In addition, quality of more than 300 recordings was inspected and showed that 77% of the data is exploitable. In depth, the percentage of sole presence of the newborn was estimated at 62% within recordings.

CONCLUSIONS

This study demonstrates that audio-video acquisitions are feasible on a large scale in real life in NICU. The experience also allowed us to make a clear observation of the requirements and challenges that will have to be overcome in order to set up audio-video monitoring methods.

Machine Learning-Based Automatic Classification of Video Recorded Neonatal Manipulations and Associated Physiological Parameters: A Feasibility Study

Our objective in this study was to determine if machine learning (ML) can automatically recognize neonatal manipulations, along with associated changes in physiological parameters. A retrospective observational study was carried out in two Neonatal Intensive Care Units (NICUs) between December 2019 to April 2020. Both the video and physiological data (heart rate (HR) and oxygen saturation (SpO₂)) were captured during NICU hospitalization. The proposed classification of neonatal manipulations was achieved by a deep learning system consisting of an Inception-v3 convolutional neural network (CNN), followed by transfer learning layers of Long Short-Term Memory (LSTM). Physiological signals prior to manipulations (baseline) were compared to during and after manipulations. The validation of the system was done using the leave-one-out strategy with input of 8 s of video exhibiting manipulation activity. Ten neonates were video recorded during an average length of stay of 24.5 days. Each neonate had an average of 528 manipulations during their NICU hospitalization, with the average duration of performing these manipulations varying from 28.9 s for patting, 45.5 s for a diaper change, and 108.9 s for tube feeding. The accuracy of the system was 95% for training and 85% for the validation dataset. In neonates <32 weeks’ gestation, diaper changes were associated with significant changes in HR and SpO₂, and, for neonates ≥32 weeks’ gestation, patting and tube feeding were associated with significant changes in HR. The presented system can classify and document the manipulations with high accuracy. Moreover, the study suggests that manipulations impact physiological parameters.

Automatic and Continuous Discomfort Detection for Premature Infants in a NICU Using Video-Based Motion Analysis

Frequent pain and discomfort in premature infants can lead to long-term adverse neurodevelopmental outcomes. Video-based monitoring is considered to be a promising contactless method for identification of discomfort moments. In this study, we propose a video-based method for automated detection of infant discomfort. The method is based on analyzing facial and body motion. Therefore, motion trajectories are estimated from frame to frame using optical flow. For each video segment, we further calculate the motion acceleration rate and extract 18 time- and frequency-domain features characterizing motion patterns. A support vector machine (SVM) classifier is then applied to video sequences to recognize infant status of comfort or discomfort. The method is evaluated using 183 video segments for 11 infants from 17 heel prick events. Experimental results show an AUC of 0.94 for discomfort detection and the average accuracy of 0.86 when combining all proposed features, which is promising for clinical use.

Automated video-based detection of nocturnal motor seizures in children

Seizure detection devices can improve epilepsy care, but wearables are not always tolerated. We previously demonstrated good performance of a real‐time video‐based algorithm for detection of nocturnal convulsive seizures in adults with learning disabilities. The algorithm calculates the relative frequency content based on the group velocity reconstruction from video‐sequence optical flow. We aim to validate the video algorithm on nocturnal motor seizures in a pediatric population. We retrospectively analyzed the algorithm performance on a database including 1661 full recorded nights of 22 children (age = 3‐17 years) with refractory epilepsy at home or in a residential care setting. The algorithm detected 118 of 125 convulsions (median sensitivity per participant = 100%, overall sensitivity = 94%, 95% confidence interval = 61%‐100%) and identified all 135 hyperkinetic seizures. Most children had no false alarms; 81 false alarms occurred in six children (median false alarm rate [FAR] per participant per night = 0 [range = 0‐0.47], overall FAR = 0.05 per night). Most false alarms (62%) were behavior‐related (eg, awake and playing in bed). Our noncontact detection algorithm reliably detects nocturnal epileptic events with only a limited number of false alarms and is suitable for real‐time use.

The yield of long-term electrocardiographic recordings in refractory focal epilepsy

Objective

To determine the incidence of clinically relevant arrhythmias in refractory focal epilepsy and to assess the potential of postictal arrhythmias as risk markers for sudden unexpected death in epilepsy (SUDEP).

Methods

We recruited people with refractory focal epilepsy without signs of ictal asystole and who had at least one focal seizure per month and implanted a loop recorder with 2‐year follow‐up. The devices automatically record arrhythmias. Subjects and caregivers were instructed to make additional peri‐ictal recordings. Clinically relevant arrhythmias were defined as asystole ≥ 6 seconds; atrial fibrillation < 55 beats per minute (bpm), or > 200 bpm and duration > 30 seconds; persistent sinus bradycardia < 40 bpm while awake; and second‐ or third‐degree atrioventricular block and ventricular tachycardia/fibrillation. We performed 12‐lead electrocardiography (ECG) and tilt table testing to identify non–seizure‐related causes of asystole.

Results

We included 49 people and accumulated 1060 months of monitoring. A total of 16 474 seizures were reported, of which 4679 were captured on ECG. No clinically relevant arrhythmias were identified. Three people had a total of 18 short‐lasting (<6 seconds) periods of asystole, resulting in an incidence of 2.91 events per 1000 patient‐months. None of these coincided with a reported seizure; one was explained by micturition syncope. Other non–clinically relevant arrhythmias included paroxysmal atrial fibrillation (n = 2), supraventricular tachycardia (n = 1), and sinus tachycardia with a right bundle branch block configuration (n = 1).

Significance

We found no clinically relevant arrhythmias in people with refractory focal epilepsy during long‐term follow‐up. The absence of postictal arrhythmias does not support the use of loop recorders in people at high SUDEP risk.

Video and audio processing in paediatrics: a review

OBJECTIVE

Video and sound acquisition and processing technologies have seen great improvements in recent decades, with many applications in the biomedical area. The aim of this paper is to review the overall state of the art of advances within these topics in paediatrics and to evaluate their potential application for monitoring in the neonatal intensive care unit (NICU).

APPROACH

For this purpose, more than 150 papers dealing with video and audio processing were reviewed. For both topics, clinical applications are described according to the considered cohorts-full-term newborns, infants and toddlers or preterm newborns. Then, processing methods are presented, in terms of data acquisition, feature extraction and characterization.

MAIN RESULTS

The paper first focuses on the exploitation of video recordings; these began to be automatically processed in the 2000s and we show that they have mainly been used to characterize infant motion. Other applications, including respiration and heart rate estimation and facial analysis, are also presented. Audio processing is then reviewed, with a focus on the analysis of crying. The first studies in this field focused on induced-pain cries and the newest ones deal with spontaneous cries; the analyses are mainly based on frequency features. Then, some papers dealing with non-cry signals are also discussed.

SIGNIFICANCE

Finally, we show that even if recent improvements in digital video and signal processing allow for increased automation of processing, the context of the NICU makes a fully automated analysis of long recordings problematic. A few proposals for overcoming some of the limitations are given.