Inter-expert and intra-expert reliability in sleep spindle scoring

A personalized semi-automatic sleep spindle detection (PSASD) framework

BACKGROUND

Sleep spindles are distinct electroencephalogram (EEG) patterns of brain activity that have been posited to play a critical role in development, learning, and neurological disorders. Manual scoring for sleep spindles is labor-intensive and tedious but could supplement automated algorithms to resolve challenges posed with either approaches alone.

NEW METHODS

A Personalized Semi-Automatic Sleep Spindle Detection (PSASD) framework was developed to combine the strength of automated detection algorithms and visual expertise of human scorers. The underlying model in the PSASD framework assumes a generative model for EEG sleep spindles as oscillatory components, optimized to EEG amplitude, with remaining signals distributed into transient and low-frequency components.

RESULTS

A single graphical user interface (GUI) allows both manual scoring of sleep spindles (model training data) and verification of automatically detected spindles. A grid search approach allows optimization of parameters to balance tradeoffs between precision and recall measures.

COMPARISON WITH EXISTING METHODS

PSASD outperformed DETOKS in F1-score by 19% and 4% on the DREAMS and P-DROWS-E datasets, respectively. It also outperformed YASA in F1-score by 25% in the P-DROWS-E dataset. Further benchmarking analysis showed that PSASD outperformed four additional widely used sleep spindle detectors in F1-score in the P-DROWS-E dataset. Titration analysis revealed that four 30-second epochs are sufficient to fine-tune the model parameters of PSASD. Associations of frequency, duration, and amplitude of detected sleep spindles matched those previously reported with automated approaches.

CONCLUSIONS

Overall, PSASD improves detection of sleep spindles in EEG data acquired from both younger healthy and older adult patient populations.

Individualized temporal patterns dominate cortical upstate and sleep depth in driving human sleep spindle timing

Sleep spindles are critical for memory consolidation and strongly linked to neurological disease and aging. Despite their significance, the relative influences of factors like sleep depth, cortical up/down states, and spindle temporal patterns on individual spindle production remain poorly understood. Moreover, spindle temporal patterns are typically ignored in favor of an average spindle rate. Here, we analyze spindle dynamics in 1008 participants from the Multi-Ethnic Study of Atherosclerosis using a point process framework. Results reveal fingerprint-like temporal patterns, characterized by a refractory period followed by a period of increased spindle activity, which are highly individualized yet consistent night-to-night. We observe increased timing variability with age and distinct gender/age differences. Strikingly, and in contrast to the prevailing notion, individualized spindle patterns are the dominant determinant of spindle timing, accounting for over 70% of the statistical deviance explained by all of the factors we assessed, surpassing the contribution of slow oscillation (SO) phase (~14%) and sleep depth (~16%). Furthermore, we show spindle/SO coupling dynamics with sleep depth are preserved across age, with a global negative shift towards the SO rising slope. These findings offer novel mechanistic insights into spindle dynamics with direct experimental implications and applications to individualized electroencephalography biomarker identification.

An approach for determining the reliability of manual and digital scoring of sleep stages

STUDY OBJECTIVES

Inter-scorer variability in sleep staging is largely due to equivocal epochs that contain features of more than one stage. We propose an approach that recognizes the existence of equivocal epochs and evaluates scorers accordingly.

METHODS

Epoch-by-epoch staging was performed on 70 polysomnograms by six qualified technologists and by a digital system (Michele Sleep Scoring [MSS]). Probability that epochs assigned the same stage by only two of the six technologists (minority score) resulted from random occurrence of two errors was calculated and found to be <5%, thereby indicating that the stage assigned is an acceptable variant for the epoch. Acceptable stages were identified in each epoch as stages assigned by at least two technologists. Percent agreement between each technologist and the other five technologists, acting as judges, was determined. Agreement was considered to exist if the stage assigned by the tested scorer was one of the acceptable stages for the epoch. Stage assigned by MSS was likewise considered in agreement if included in the acceptable stages made by the technologists.

RESULTS

Agreement of technologists tested against five qualified judges increased from 80.8% (range 70.5%-86.4% among technologists) when using the majority rule, to 96.1 (89.8%-98.5%) by the proposed approach. Agreement between unedited MSS and same judges was 90.0% and increased to 92.1% after brief editing.

CONCLUSIONS

Accounting for equivocal epochs provides a more accurate estimate of a scorer's (human or digital) competence in scoring sleep stages and reduces inter-scorer disagreements. The proposed approach can be implemented in sleep-scoring training and accreditation programs.

Switching state-space modeling of neural signal dynamics

Linear parametric state-space models are a ubiquitous tool for analyzing neural time series data, providing a way to characterize the underlying brain dynamics with much greater statistical efficiency than non-parametric data analysis approaches. However, neural time series data are frequently time-varying, exhibiting rapid changes in dynamics, with transient activity that is often the key feature of interest in the data. Stationary methods can be adapted to time-varying scenarios by employing fixed-duration windows under an assumption of quasi-stationarity. But time-varying dynamics can be explicitly modeled by switching state-space models, i.e., by using a pool of state-space models with different dynamics selected by a probabilistic switching process. Unfortunately, exact solutions for state inference and parameter learning with switching state-space models are intractable. Here we revisit a switching state-space model inference approach first proposed by Ghahramani and Hinton. We provide explicit derivations for solving the inference problem iteratively after applying a variational approximation on the joint posterior of the hidden states and the switching process. We introduce a novel initialization procedure using an efficient leave-one-out strategy to compare among candidate models, which significantly improves performance compared to the existing method that relies on deterministic annealing. We then utilize this state inference solution within a generalized expectation-maximization algorithm to estimate model parameters of the switching process and the linear state-space models with dynamics potentially shared among candidate models. We perform extensive simulations under different settings to benchmark performance against existing switching inference methods and further validate the robustness of our switching inference solution outside the generative switching model class. Finally, we demonstrate the utility of our method for sleep spindle detection in real recordings, showing how switching state-space models can be used to detect and extract transient spindles from human sleep electroencephalograms in an unsupervised manner.

Hierarchical fusion detection algorithm for sleep spindle detection

Background

Sleep spindles are a vital sign implying that human beings have entered the second stage of sleep. In addition, they can effectively reflect a person’s learning and memory ability, and clinical research has shown that their quantity and density are crucial markers of brain function. The “gold standard” of spindle detection is based on expert experience; however, the detection cost is high, and the detection time is long. Additionally, the accuracy of detection is influenced by subjectivity.

Methods

To improve detection accuracy and speed, reduce the cost, and improve efficiency, this paper proposes a layered spindle detection algorithm. The first layer used the Morlet wavelet and RMS method to detect spindles, and the second layer employed an improved k-means algorithm to improve spindle detection efficiency. The fusion algorithm was compared with other spindle detection algorithms to prove its effectiveness.

Results

The hierarchical fusion spindle detection algorithm showed good performance stability, and the fluctuation range of detection accuracy was minimal. The average value of precision was 91.6%, at least five percentage points higher than other methods. The average value of recall could reach 89.1%, and the average value of specificity was close to 95%. The mean values of accuracy and F1-score in the subject sample data were 90.4 and 90.3%, respectively. Compared with other methods, the method proposed in this paper achieved significant improvement in terms of precision, recall, specificity, accuracy, and F1-score.

Conclusion

A spindle detection method with high steady-state accuracy and fast detection speed is proposed, which combines the Morlet wavelet with window RMS and an improved k-means algorithm. This method provides a powerful tool for the automatic detection of spindles and improves the efficiency of spindle detection. Through simulation experiments, the sampled data were analyzed and verified to prove the feasibility and effectiveness of this method.

[Sleep spindles-Function, detection and use as biomarker for diagnostics in psychiatry]

BACKGROUND

The sleep spindle is a graphoelement of an electroencephalogram (EEG), which can be observed in light and deep sleep. Alterations in spindle activity have been described for a range of psychiatric disorders. Due to their relatively constant properties, sleep spindles may therefore be potential biomarkers in psychiatric diagnostics.

METHOD

This article presents an overview of the state of the science on the characteristics and functions of the sleep spindle as well as known alterations of spindle activity in psychiatric disorders. Various methodological approaches and developments of spindle detection are explained with respect to their potential for application in psychiatric diagnostics.

RESULTS AND CONCLUSION

Although alterations in spindle activity in psychiatric disorders are known and have been described in detail, their exact potential for psychiatric diagnostics has yet to be fully determined. In this respect, the acquisition of knowledge in research is currently constrained by manual and automated methods for spindle detection, which require high levels of resources and are error prone. Newer approaches to spindle detection based on deep-learning procedures could overcome the difficulties of previous detection methods, and thus open up new possibilities for the practical application of sleep spindles as biomarkers in the psychiatric practice.

Advanced sleep spindle identification with neural networks

Sleep spindles are neurophysiological phenomena that appear to be linked to memory formation and other functions of the central nervous system, and that can be observed in electroencephalographic recordings (EEG) during sleep. Manually identified spindle annotations in EEG recordings suffer from substantial intra- and inter-rater variability, even if raters have been highly trained, which reduces the reliability of spindle measures as a research and diagnostic tool. The Massive Online Data Annotation (MODA) project has recently addressed this problem by forming a consensus from multiple such rating experts, thus providing a corpus of spindle annotations of enhanced quality. Based on this dataset, we present a U-Net-type deep neural network model to automatically detect sleep spindles. Our model’s performance exceeds that of the state-of-the-art detector and of most experts in the MODA dataset. We observed improved detection accuracy in subjects of all ages, including older individuals whose spindles are particularly challenging to detect reliably. Our results underline the potential of automated methods to do repetitive cumbersome tasks with super-human performance.

Towards Sleep Study Automation: Detection Evaluation of Respiratory-Related Events

The diagnosis of sleep disordered breathing depends on the detection of several respiratory-related events: apneas, hypopneas, snores, or respiratory event-related arousals from sleep studies. While a number of automatic detection methods have been proposed, reproducibility of these methods has been an issue, in part due to the absence of a generally accepted protocol for evaluating their results. With sleep measurements this is usually treated as a classification problem and the accompanying issue of localization is not treated as similarly critical. To address these problems we present a detection evaluation protocol that is able to qualitatively assess the match between two annotations of respiratory-related events. This protocol relies on measuring the relative temporal overlap between two annotations in order to find an alignment that maximizes their F1-score at the sequence level. This protocol can be used in applications which require a precise estimate of the number of events, total event duration, and a joint estimate of event number and duration. We assess its application using a data set that contains over 10,000 manually annotated snore events from 9 subjects, and show that when using the American Academy of Sleep Medicine Manual standard, two sleep technologists can achieve an F1-score of 0.88 when identifying the presence of snore events. In addition, we drafted rules for marking snore boundaries and showed that one sleep technologist can achieve F1-score of 0.94 at the same tasks. Finally, we compared our protocol against the protocol that is used to evaluate sleep spindle detection and highlighted the differences.

Effect of a Recliner Chair with Rocking Motions on Sleep Efficiency

In this study, we analyze the effect of a recliner chair with rocking motions on sleep quality of naps using automated sleep scoring and spindle detection models. The quality of sleep corresponding to the two rocking motions was measured quantitatively and qualitatively. For the quantitative evaluation, we conducted a sleep parameter analysis based on the results of the estimated sleep stages obtained on the brainwave and spindle estimation, and a sleep survey assessment from the participants was analyzed for the qualitative evaluation. The analysis showed that sleep in the recliner chair with rocking motions positively increased the duration of the spindles and deep sleep stage, resulting in improved sleep quality.

Scalp EEG markers of normal infant development using visual and computational approaches

The infant brain is rapidly developing, and these changes are reflected in scalp electroencephalography (EEG) features, including power spectrum and sleep spindle characteristics. These biomarkers not only mirror infant development, but they are also altered by conditions such as epilepsy, autism, developmental delay, and trisomy 21. Prior studies of early development were generally limited by small cohort sizes, lack of a specific focus on infancy (0-2 years), and exclusive use of visual marking for sleep spindles. Therefore, we measured the EEG power spectrum and sleep spindles in 240 infants ranging from 0-24 months. To rigorously assess these metrics, we used both clinical visual assessment and computational techniques, including automated sleep spindle detection. We found that the peak frequency and power of the posterior dominant rhythm (PDR) increased with age, and a corresponding peak occurred in the EEG power spectra. Based on both clinical and computational measures, spindle duration decreased with age, and spindle synchrony increased with age. Our novel metric of spindle asymmetry suggested that peak spindle asymmetry occurs at 6-9 months of age.Clinical Relevance- Here we provide a robust characterization of the development of EEG brain rhythms during infancy. This can be used as a basis of comparison for studies of infant neurological disease, including epilepsy, autism, developmental delay, and trisomy 21.

Hippocampal spindles and barques are normal intracranial electroencephalographic entities

OBJECTIVE

To assess whether hippocampal spindles and barques are markers of epileptogenicity.

METHODS

Focal epilepsy patients that underwent stereo-electroencephalography implantation with at least one electrode in their hippocampus were selected (n = 75). The occurrence of spindles and barques in the hippocampus was evaluated in each patient. We created pairs of pathologic and pathology-free groups according to two sets of criteria: 1. Non-invasive diagnostic criteria (patients grouped according to focal epilepsy classification). 2. Intracranial neurophysiological criteria (patient's hippocampi grouped according to their seizure onset involvement).

RESULTS

Hippocampal spindles and barques appear equally often in both pathologic and pathology-free groups, both for non-invasive (P_spindles = 0.73; P_barques = 0.46) and intracranial criteria (P_spindles = 0.08; P_barques = 0.26). In Engel Class I patients, spindles occurred with similar incidence both within the non-invasive (P = 0.67) and the intracranial criteria group (P = 0.20). Barques were significantly more frequent in extra-temporal lobe epilepsy defined by either non-invasive (P = 0.01) or intracranial (P = 0.01) criteria.

CONCLUSIONS

Both spindles and barques are normal entities of the hippocampal intracranial electroencephalogram. The presence of barques may also signify lack of epileptogenic properties in the hippocampus.

SIGNIFICANCE

Understanding that hippocampal spindles and barques do not reflect epileptogenicity is critical for correct interpretation of epilepsy surgery evaluations and appropriate surgical treatment selection.

Sleep spindles comprise a subset of a broader class of electroencephalogram events

STUDY OBJECTIVES

Sleep spindles are defined based on expert observations of waveform features in the electroencephalogram (EEG) traces. This is a potentially limiting characterization, as transient oscillatory bursts like spindles are easily obscured in the time domain by higher amplitude activity at other frequencies or by noise. It is therefore highly plausible that many relevant events are missed by current approaches based on traditionally defined spindles. Given their oscillatory structure, we reexamine spindle activity from first principles, using time-frequency activity in comparison to scored spindles.

METHODS

Using multitaper spectral analysis, we observe clear time-frequency peaks in the sigma (10-16 Hz) range (TFσ peaks). While nearly every scored spindle coincides with a TFσ peak, numerous similar TFσ peaks remain undetected. We therefore perform statistical analyses of spindles and TFσ peaks using manual and automated detection methods, comparing event cooccurrence, morphological similarities, and night-to-night consistency across multiple datasets.

RESULTS

On average, TFσ peaks have more than three times the rate of spindles (mean rate: 9.8 vs. 3.1 events/minute). Moreover, spindles subsample the most prominent TFσ peaks with otherwise identical spectral morphology. We further demonstrate that detected TFσ peaks have stronger night-to-night rate stability (ρ = 0.98) than spindles (ρ = 0.67), while covarying with spindle rates across subjects (ρ = 0.72).

CONCLUSIONS

These results provide compelling evidence that traditionally defined spindles constitute a subset of a more generalized class of EEG events. TFσ peaks are therefore a more complete representation of the underlying phenomenon, providing a more consistent and robust basis for future experiments and analyses.

Heart Rate and Heart Rate Variability Change with Sleep Stage in Dairy Cows

Changes to the amount and patterns of sleep stages could be a useful tool to assess the effects of stress or changes to the environment in animal welfare research. However, the gold standard method, polysomnography PSG, is difficult to use with large animals such as dairy cows. Heart rate (HR) and heart rate variability (HRV) can be used to predict sleep stages in humans and could be useful as an easier method to identify sleep stages in cows. We compared the mean HR and HRV and lying posture of dairy cows at pasture and when housed, with sleep stages identified through PSG. HR and HRV were higher when cows were moving their heads or when lying flat on their side. Overall, mean HR decreased with depth of sleep. There was more variability in time between successive heart beats during REM sleep, and more variability in time between heart beats when cows were awake and in REM sleep. These shifts in HR measures between sleep stages followed similar patterns despite differences in mean HR between the groups. Our results show that HR and HRV measures could be a promising alternative method to PSG for assessing sleep in dairy cows.

Large-scale assessment of consistency in sleep stage scoring rules among multiple sleep centers using an interpretable machine learning algorithm

STUDY OBJECTIVES

Polysomnography is the gold standard in identifying sleep stages; however, there are discrepancies in how technicians use the standards. Because organizing meetings to evaluate this discrepancy and/or reach a consensus among multiple sleep centers is time-consuming, we developed an artificial intelligence system to efficiently evaluate the reliability and consistency of sleep scoring and hence the sleep center quality.

METHODS

An interpretable machine learning algorithm was used to evaluate the interrater reliability (IRR) of sleep stage annotation among sleep centers. The artificial intelligence system was trained to learn raters from 1 hospital and was applied to patients from the same or other hospitals. The results were compared with the experts' annotation to determine IRR. Intracenter and intercenter assessments were conducted on 679 patients without sleep apnea from 6 sleep centers in Taiwan. Centers with potential quality issues were identified by the estimated IRR.

RESULTS

In the intracenter assessment, the median accuracy ranged from 80.3%-83.3%, with the exception of 1 hospital, which had an accuracy of 72.3%. In the intercenter assessment, the median accuracy ranged from 75.7%-83.3% when the 1 hospital was excluded from testing and training. The performance of the proposed method was higher for the N2, awake, and REM sleep stages than for the N1 and N3 stages. The significant IRR discrepancy of the 1 hospital suggested a quality issue. This quality issue was confirmed by the physicians in charge of the 1 hospital.

CONCLUSIONS

The proposed artificial intelligence system proved effective in assessing IRR and hence the sleep center quality.

Machine learning prediction of sleep stages in dairy cows from heart rate and muscle activity measures

Sleep is important for cow health and shows promise as a tool for assessing welfare, but methods to accurately distinguish between important sleep stages are difficult and impractical to use with cattle in typical farm environments. The objective of this study was to determine if data from more easily applied non-invasive devices assessing neck muscle activity and heart rate (HR) alone could be used to differentiate between sleep stages. We developed, trained, and compared two machine learning models using neural networks and random forest algorithms to predict sleep stages from 15 variables (features) of the muscle activity and HR data collected from 12 cows in two environments. Using k-fold cross validation we compared the success of the models to the gold standard, Polysomnography (PSG). Overall, both models learned from the data and were able to accurately predict sleep stages from HR and muscle activity alone with classification accuracy in the range of similar human models. Further research is required to validate the models with a larger sample size, but the proposed methodology appears to give an accurate representation of sleep stages in cattle and could consequentially enable future sleep research into conditions affecting cow sleep and welfare.

Analysis and visualization of sleep stages based on deep neural networks

Automatic sleep stage scoring based on deep neural networks has come into focus of sleep researchers and physicians, as a reliable method able to objectively classify sleep stages would save human resources and simplify clinical routines. Due to novel open-source software libraries for machine learning, in combination with enormous recent progress in hardware development, a paradigm shift in the field of sleep research towards automatic diagnostics might be imminent. We argue that modern machine learning techniques are not just a tool to perform automatic sleep stage classification, but are also a creative approach to find hidden properties of sleep physiology. We have already developed and established algorithms to visualize and cluster EEG data, facilitating first assessments on sleep health in terms of sleep-apnea and consequently reduced daytime vigilance. In the following study, we further analyze cortical activity during sleep by determining the probabilities of momentary sleep stages, represented as hypnodensity graphs and then computing vectorial cross-correlations of different EEG channels. We can show that this measure serves to estimate the period length of sleep cycles and thus can help to find disturbances due to pathological conditions.

Validation of an Automatic Arousal Detection Algorithm for Whole-Night Sleep EEG Recordings

Arousals during sleep are transient accelerations of the EEG signal, considered to reflect sleep perturbations associated with poorer sleep quality. They are typically detected by visual inspection, which is time consuming, subjective, and prevents good comparability across scorers, studies and research centres. We developed a fully automatic algorithm which aims at detecting artefact and arousal events in whole-night EEG recordings, based on time-frequency analysis with adapted thresholds derived from individual data. We ran an automated detection of arousals over 35 sleep EEG recordings in healthy young and older individuals and compared it against human visual detection from two research centres with the aim to evaluate the algorithm performance. Comparison across human scorers revealed a high variability in the number of detected arousals, which was always lower than the number detected automatically. Despite indexing more events, automatic detection showed high agreement with human detection as reflected by its correlation with human raters and very good Cohen's kappa values. Finally, the sex of participants and sleep stage did not influence performance, while age may impact automatic detection, depending on the human rater considered as gold standard. We propose our freely available algorithm as a reliable and time-sparing alternative to visual detection of arousals.

Association of sleep spindle characteristics with executive functioning in healthy sedentary middle-aged and older adults

To determine the relationship between sleep spindle characteristics (density, power and frequency), executive functioning and cognitive decline in older adults, we studied a convenience subsample of healthy middle-aged and older participants of the Brain in Motion study. Participants underwent a single night of unattended in-home polysomnography with neurocognitive testing carried out shortly afterwards. Spectral analysis of the EEG was performed to derive spindle characteristics in both central and frontal derivations during non-rapid eye movement (NREM) Stage 2 and 3. Multiple linear regressions were used to examine associations between spindle characteristics and cognitive outcomes, with age, body mass index (BMI), periodic limb movements index (PLMI) and apnea hypopnea index (AHI) as covariates. NREM Stage 2 total spindle density was significantly associated with executive functioning (central: β = .363, p = .016; frontal: β = .408, p = .004). NREM Stage 2 fast spindle density was associated with executive functioning (central: β = .351, p = .022; frontal: β = .380, p = .009) and Montreal Cognitive Assessment score (MoCA, central: β = .285, p = .037; frontal: β = .279, p = .032). NREM Stage 2 spindle frequency was also associated with MoCA score (central: β = .337, p = .013). Greater spindle density and fast spindle density were associated with better executive functioning and less cognitive decline in our study population. Our cross-sectional design cannot infer causality. Longitudinal studies will be required to assess the ability of spindle characteristics to predict future cognitive status.

Sleep Spindle Detection Using RUSBoost and Synchrosqueezed Wavelet Transform

Sleep spindles are important electroencephalographic (EEG) waveforms in sleep medicine; however, it is burdensome even for experts to detect spindles, so automatic spindle detection methodologies have been investigated. Conventional methods utilize waveforms template matching or machine learning for detecting spindles. In the former approach, it is necessary to tune thresholds for individual adaptation, while the latter approach has the problem of imbalanced data because the amount of sleep spindles is small compared with the entire EEG data. The present work proposes a sleep spindle detection method that combines wavelet synchrosqueezed transform (SST) and random under-sampling boosting (RUSBoost). SST is a time-frequency analysis method suitable for extracting features of spindle waveforms. RUSBoost is a framework for coping with the imbalanced data problem. The proposed SST-RUS can deal with the imbalanced data in spindle detection and does not require threshold tuning because RUSBoost uses majority voting of weak classifiers for discrimination. The performance of SST-RUS was validated using an open-access database called the Montreal archives of sleep studies cohort 1 (MASS-C1), which showed an F-measure of 0.70 with a sensitivity of 76.9% and a positive predictive value of 61.2%. The proposed method can reduce the burden of PSG scoring.

Sleep Spindles: Mechanisms and Functions

Sleep spindles are burstlike signals in the electroencephalogram (EEG) of the sleeping mammalian brain and electrical surface correlates of neuronal oscillations in thalamus. As one of the most inheritable sleep EEG signatures, sleep spindles probably reflect the strength and malleability of thalamocortical circuits that underlie individual cognitive profiles. We review the characteristics, organization, regulation, and origins of sleep spindles and their implication in non-rapid-eye-movement sleep (NREMS) and its functions, focusing on human and rodent. Spatially, sleep spindle-related neuronal activity appears on scales ranging from small thalamic circuits to functional cortical areas, and generates a cortical state favoring intracortical plasticity while limiting cortical output. Temporally, sleep spindles are discrete events, part of a continuous power band, and elements grouped on an infraslow time scale over which NREMS alternates between continuity and fragility. We synthesize diverse and seemingly unlinked functions of sleep spindles for sleep architecture, sensory processing, synaptic plasticity, memory formation, and cognitive abilities into a unifying sleep spindle concept, according to which sleep spindles 1) generate neural conditions of large-scale functional connectivity and plasticity that outlast their appearance as discrete EEG events, 2) appear preferentially in thalamic circuits engaged in learning and attention-based experience during wakefulness, and 3) enable a selective reactivation and routing of wake-instated neuronal traces between brain areas such as hippocampus and cortex. Their fine spatiotemporal organization reflects NREMS as a physiological state coordinated over brain and body and may indicate, if not anticipate and ultimately differentiate, pathologies in sleep and neurodevelopmental, -degenerative, and -psychiatric conditions.

Validation of 'Somnivore', a Machine Learning Algorithm for Automated Scoring and Analysis of Polysomnography Data

Manual scoring of polysomnography data is labor-intensive and time-consuming, and most existing software does not account for subjective differences and user variability. Therefore, we evaluated a supervised machine learning algorithm, Somnivore^TM, for automated wake–sleep stage classification. We designed an algorithm that extracts features from various input channels, following a brief session of manual scoring, and provides automated wake-sleep stage classification for each recording. For algorithm validation, polysomnography data was obtained from independent laboratories, and include normal, cognitively-impaired, and alcohol-treated human subjects (total n = 52), narcoleptic mice and drug-treated rats (total n = 56), and pigeons (n = 5). Training and testing sets for validation were previously scored manually by 1–2 trained sleep technologists from each laboratory. F-measure was used to assess precision and sensitivity for statistical analysis of classifier output and human scorer agreement. The algorithm gave high concordance with manual visual scoring across all human data (wake 0.91 ± 0.01; N1 0.57 ± 0.01; N2 0.81 ± 0.01; N3 0.86 ± 0.01; REM 0.87 ± 0.01), which was comparable to manual inter-scorer agreement on all stages. Similarly, high concordance was observed across all rodent (wake 0.95 ± 0.01; NREM 0.94 ± 0.01; REM 0.91 ± 0.01) and pigeon (wake 0.96 ± 0.006; NREM 0.97 ± 0.01; REM 0.86 ± 0.02) data. Effects of classifier learning from single signal inputs, simple stage reclassification, automated removal of transition epochs, and training set size were also examined. In summary, we have developed a polysomnography analysis program for automated sleep-stage classification of data from diverse species. Somnivore enables flexible, accurate, and high-throughput analysis of experimental and clinical sleep studies.

Ground truth construction and parameter tuning for the detection of sleep spindle timing in rodents

BACKGROUND

The precise detection of cortical sleep spindles is critical to basic research on memory consolidation in rodents. Previous research using automatic spindle detection algorithms often lacks systematic parameter variations and validations.

NEW METHOD

We present a method to systematically tune and validate algorithm parameters in automatic spindle detection algorithms using a moderate number of human raters.

RESULTS

Comparing a Hilbert transform-based algorithm to a ground truth constructed by six human raters, this method produced a parameter set yielding an F1 score of 0.82 at 10 ms resolution. The algorithm performance fell within the range of human agreement with the ground truth. Both human and algorithm failures arose largely from disagreement in spindle boundaries rather than spindle occurrence. With no additional tuning, the algorithm performed similarly in recordings from different days or rats.

COMPARISON WITH EXISTING METHODS

Most spindle detection algorithms do not perform systematic parameter variations and validation using a ground truth. To our knowledge, our study is the first in which rodent spindle data is scored by humans, and in which an automatic spindle detection algorithm is evaluated with respect to this ground truth. The rodent data from this study make it possible to compare our algorithm with others previously tested on human data.

CONCLUSIONS

We present a general ground truth based approach for the tuning and validation of spindle extraction algorithms and suggest that algorithms aimed at extracting precise spindle timing in rats should use a systematic approach for parameter tuning.

Tongue Image Database Construction Based on the Expert Opinions: Assessment for Individual Agreement and Methods for Expert Selection

This study aims at introducing a method for individual agreement evaluation to identify the discordant raters from the experts' group. We exclude those experts and decide the best experts selection method, so as to improve the reliability of the constructed tongue image database based on experts' opinions. Fifty experienced experts from the TCM diagnostic field all over China were invited to give ratings for 300 randomly selected tongue images. Gwet's AC₁ (first-order agreement coefficient) was used to calculate the interrater and intrarater agreement. The optimization of the interrater agreement and the disagreement score were put forward to evaluate the external consistency for individual expert. The proposed method could successfully optimize the interrater agreement. By comparing three experts' selection methods, the interrater agreement was, respectively, increased from 0.53 [0.32-0.75] for original one to 0.64 [0.39-0.80] using method A (inclusion of experts whose intrarater agreement>0.6), 0.69 [0.63-0.81] using method B (inclusion of experts whose disagreement score=“0”), and 0.76 [0.67-0.83] using method C (inclusion of experts whose intrarater agreement>0.6& disagreement score=“0”). In this study, we provide an estimate of external consistency for individual expert, and the comprehensive consideration of both the internal consistency and the external consistency for each expert would be superior to either one in the tongue image construction based on expert opinions.

A sleep spindle detection algorithm that emulates human expert spindle scoring

BACKGROUND

Sleep spindles are a marker of stage 2 NREM sleep that are linked to learning & memory and are altered by many neurological diseases. Although visual inspection of the EEG is considered the gold standard for spindle detection, it is time-consuming, costly and can introduce inter/ra-scorer bias.

NEW METHOD

Our goal was to develop a simple and efficient sleep-spindle detector (algorithm #7, or 'A7') that emulates human scoring. 'A7' runs on a single EEG channel and relies on four parameters: the absolute sigma power, relative sigma power, and correlation/covariance of the sigma band-passed signal to the original EEG signal. To test the performance of the detector, we compared it against a gold standard spindle dataset derived from the consensus of a group of human experts.

RESULTS

The by-event performance of the 'A7' spindle detector was 74% precision, 68% recall (sensitivity), and an F1-score of 0.70. This performance was equivalent to an individual human expert (average F1-score = 0.67).

COMPARISON WITH EXISTING METHOD(S)

The F1-score of 'A7' was 0.17 points higher than other spindle detectors tested. Existing detectors have a tendency to find large numbers of false positives compared to human scorers. On a by-subject basis, the spindle density estimates produced by A7 were well correlated with human experts (r² = 0.82) compared to the existing detectors (average r² = 0.27).

CONCLUSIONS

The 'A7' detector is a sensitive and precise tool designed to emulate human spindle scoring by minimizing the number of 'hidden spindles' detected. We provide an open-source implementation of this detector for further use and testing.

Admission EEG findings in diverse paediatric cerebral malaria populations predict outcomes

BACKGROUND

Electroencephalography at hospital presentation may offer important insights regarding prognosis that can inform understanding of cerebral malaria (CM) pathophysiology and potentially guide patient selection and risk stratification for future clinical trials. Electroencephalogram (EEG) findings in children with CM in Uganda and Malawi were compared and associations between admission EEG findings and outcome across this diverse population were assessed. Demographic, clinical and admission EEG data from Ugandan and Malawian children admitted from 2009 to 2012 with CM were gathered, and survivors assessed for neurological abnormalities at discharge.

RESULTS

281 children were enrolled (Uganda n = 122, Malawi n = 159). The Malawian population was comprised only of retinopathy positive children (versus 72.5% retinopathy positive in Uganda) and were older (4.2 versus 3.7 years; p = 0.046), had a higher HIV prevalence (9.0 versus 2.8%; p = 0.042), and worse hyperlactataemia (7.4 versus 5.2 mmol/L; p < 0.001) on admission compared to the Ugandan children. EEG findings differed between the two groups in terms of average voltage and frequencies, reactivity, asymmetry, and the presence/absence of sleep architecture. In univariate analyses pooling EEG and outcomes data for both sites, higher average and maximum voltages, faster dominant frequencies, and retained reactivity were associated with survival (all p < 0.05). Focal slowing was associated with death (OR 2.93; 95% CI 1.77-7.30) and a lower average voltage was associated with neurological morbidity in survivors (p = 0.0032).

CONCLUSIONS

Despite substantial demographic and clinical heterogeneity between subjects in Malawi and Uganda as well as different EEG readers at each site, EEG findings on admission predicted mortality and morbidity. For CM clinical trials aimed at decreasing mortality or morbidity, EEG may be valuable for risk stratification and/or subject selection.

Analysis of Multichannel EEG Patterns During Human Sleep: A Novel Approach

Classic visual sleep stage scoring is based on electroencephalogram (EEG) frequency band analysis of 30 s epochs and is commonly performed by highly trained medical sleep specialists using additional information from submental EMG and eye movements electrooculogram (EOG). In this study, we provide the proof-of-principle in 40 subjects that sleep stages can be consistently differentiated solely on the basis of spatial 3-channel EEG patterns based on root-mean-square (RMS) amplitudes. The polysomnographic 3-channel EEG data are pre-processed by RMS averaging over intervals of 30 s leading to spatial cortical activity patterns represented by 3-dimensional vectors. These patterns are visualized using multidimensional scaling (MDS), allowing a comparison of the spatial cortical activity patterns with the conventional visual sleep scoring system according to the American Academy of Sleep Medicine (AASM). Spatial cortical activity patterns based on RMS amplitudes naturally divide into different clusters that correspond to visually scored sleep stages. Furthermore, these clusters are reproducible between different subjects. Especially the cluster associated with the REM sleep stage seems to be very different from the one associated with the wake state. This study provides a proof-of-principle that it is possible to separate sleep stages solely by analyzing spatially distributed EEG RMS amplitudes reflecting cortical activity and without classical EEG feature extractions like power spectrum analysis.

Reliability of the American Academy of Sleep Medicine Rules for Assessing Sleep Depth in Clinical Practice

STUDY OBJECTIVES

The American Academy of Sleep Medicine has published manuals for scoring polysomnograms that recommend time spent in non-rapid eye movement sleep stages (stage N1, N2, and N3 sleep) be reported. Given the well-established large interrater variability in scoring stage N1 and N3 sleep, we determined the range of time in stage N1 and N3 sleep scored by a large number of technologists when compared to reasonably estimated true values.

METHODS

Polysomnograms of 70 females were scored by 10 highly trained sleep technologists, two each from five different academic sleep laboratories. Range and confidence interval (CI = difference between the 5th and 95th percentiles) of the 10 times spent in stage N1 and N3 sleep assigned in each polysomnogram were determined. Average values of times spent in stage N1 and N3 sleep generated by the 10 technologists in each polysomnogram were considered representative of the true values for the individual polysomnogram. Accuracy of different technologists in estimating delta wave duration was determined by comparing their scores to digitally determined durations.

RESULTS

The CI range of the ten N1 scores was 4 to 39 percent of total sleep time (% TST) in different polysomnograms (mean CI ± standard deviation = 11.1 ± 7.1 % TST). Corresponding range for N3 was 1 to 28 % TST (14.4 ± 6.1 % TST). For stage N1 and N3 sleep, very low or very high values were reported for virtually all polysomnograms by different technologists. Technologists varied widely in their assignment of stage N3 sleep, scoring that stage when the digitally determined time of delta waves ranged from 3 to 17 seconds.

CONCLUSIONS

Manual scoring of non-rapid eye movement sleep stages is highly unreliable among highly trained, experienced technologists. Measures of sleep continuity and depth that are reliable and clinically relevant should be a focus of clinical research.

The Accuracy, Night-to-Night Variability, and Stability of Frontopolar Sleep Electroencephalography Biomarkers

STUDY OBJECTIVES

To assess the validity of sleep architecture and sleep continuity biomarkers obtained from a portable, multichannel forehead electroencephalography (EEG) recorder.

METHODS

Forty-seven subjects simultaneously underwent polysomnography (PSG) while wearing a multichannel frontopolar EEG recording device (Sleep Profiler). The PSG recordings independently staged by 5 registered polysomnographic technologists were compared for agreement with the autoscored sleep EEG before and after expert review. To assess the night-to-night variability and first night bias, 2 nights of self-applied, in-home EEG recordings obtained from a clinical cohort of 63 patients were used (41% with a diagnosis of insomnia/depression, 35% with insomnia/obstructive sleep apnea, and 17.5% with all three). The between-night stability of abnormal sleep biomarkers was determined by comparing each night's data to normative reference values.

RESULTS

The mean overall interscorer agreements between the 5 technologists were 75.9%, and the mean kappa score was 0.70. After visual review, the mean kappa score between the autostaging and five raters was 0.67, and staging agreed with a majority of scorers in at least 80% of the epochs for all stages except stage N1. Sleep spindles, autonomic activation, and stage N3 exhibited the least between-night variability (P < .0001) and strongest between-night stability. Antihypertensive medications were found to have a significant effect on sleep quality biomarkers (P < .02).

CONCLUSIONS

A strong agreement was observed between the automated sleep staging and human-scored PSG. One night's recording appeared sufficient to characterize abnormal slow wave sleep, sleep spindle activity, and heart rate variability in patients, but a 2-night average improved the assessment of all other sleep biomarkers.

COMMENTARY

Two commentaries on this article appear in this issue on pages 771 and 773.

Sleep spindle detection based on non-experts: A validation study

Accurate and efficient detection of sleep spindles is a methodological challenge. The present study describes a method of using non-experts for manual detection of sleep spindles. We recruited five experts and 168 non-experts to manually identify spindles in stage N2 and stage N3 sleep data using a MATLAB interface. Scorers classified each spindle into definite and indefinite spindle (with weights of 1 and 0.5, respectively). First, a method of optimizing the thresholds of the expert/non-expert group consensus according to the results of experts and non-experts themselves is described. Using this method, we established expert and non-expert group standards from expert and non-expert scorers, respectively, and evaluated the performance of the non-expert group standards by compared with the expert group standard (termed EGS). The results indicated that the highest performance was the non-expert group standard when definite spindles were only considered (termed nEGS-1; F1 score = 0.78 for N2; 0.68 for N3). Second, four automatic spindle detection methods were compared with the EGS. We found that the performance of nEGS-1 versus EGS was higher than that of the four automated methods. Our results also showed positive correlation between the mean F1 score of individual expert in EGS and the F1 score of nEGS-1 versus EGS across 30 segments of stage N2 data (r = 0.61, P < 0.001). Further, we found that six and nine non-experts were needed to manually identify spindles in stages N2 and N3, respectively, while maintaining acceptable performance of nEGS-1 versus EGS (F1 score = 0.79 for N2; 0.64 for N3). In conclusion, this study establishes a detailed process for detection of sleep spindles by non-experts in a crowdsourcing scheme.

Night sleep in patients with vegetative state

Polysomnographic recording of night sleep was carried out in 15 patients with the diagnosis vegetative state (syn. unresponsive wakefulness syndrome). Sleep scoring was performed by three raters, and confirmed by means of a spectral power analysis of the electroencephalogram, electrooculogram and electromyogram. All patients but one exhibited at least some signs of sleep. In particular, sleep stage N1 was found in 13 patients, N2 in 14 patients, N3 in nine patients, and rapid eye movement sleep in 10 patients. Three patients exhibited all phenomena characteristic for normal sleep, including spindles and rapid eye movements. However, in all but one patient, sleep patterns were severely disturbed as compared with normative data. All patients had frequent and long periods of wakefulness during the night. In some apparent rapid eye movement sleep episodes, no eye movements were recorded. Sleep spindles were detected in five patients only, and their density was very low. We conclude that the majority of vegetative state patients retain some important circadian changes. Further studies are necessary to disentangle multiple factors potentially affecting sleep pattern of vegetative state patients.

Minimizing Interrater Variability in Staging Sleep by Use of Computer-Derived Features

STUDY OBJECTIVES

Inter-scorer variability in sleep staging of polysomnograms (PSGs) results primarily from difficulty in determining whether: (1) an electroencephalogram pattern of wakefulness spans > 15 sec in transitional epochs, (2) spindles or K complexes are present, and (3) duration of delta waves exceeds 6 sec in a 30-sec epoch. We hypothesized that providing digitally derived information about these variables to PSG scorers may reduce inter-scorer variability.

METHODS

Fifty-six PSGs were scored (five-stage) by two experienced technologists, (first manual, M1). Months later, the technologists edited their own scoring (second manual, M2). PSGs were then scored with an automatic system and the same two technologists and an additional experienced technologist edited them, epoch-by-epoch (Edited-Auto). This resulted in seven manual scores for each PSG. The two M2 scores were then independently modified using digitally obtained values for sleep depth and delta duration and digitally identified spindles and K complexes.

RESULTS

Percent agreement between scorers in M2 was 78.9 ± 9.0% before modification and 96.5 ± 2.6% after. Errors of this approach were defined as a change in a manual score to a stage that was not assigned by any scorer during the seven manual scoring sessions. Total errors averaged 7.1 ± 3.7% and 6.9 ± 3.8% of epochs for scorers 1 and 2, respectively, and there was excellent agreement between the modified score and the initial manual score of each technologist.

CONCLUSIONS

Providing digitally obtained information about sleep depth, delta duration, spindles and K complexes during manual scoring can greatly reduce interrater variability in sleep staging by eliminating the guesswork in scoring epochs with equivocal features.

Automatic Sleep Spindle Detection and Genetic Influence Estimation Using Continuous Wavelet Transform

Mounting evidence for the role of sleep spindles in neuroplasticity has led to an increased interest in these non-rapid eye movement (NREM) sleep oscillations. It has been hypothesized that fast and slow spindles might play a different role in memory processing. Here, we present a new sleep spindle detection algorithm utilizing a continuous wavelet transform (CWT) and individual adjustment of slow and fast spindle frequency ranges. Eighteen nap recordings of ten subjects were used for algorithm validation. Our method was compared with both a human scorer and a commercially available SIESTA spindle detector. For the validation set, mean agreement between our detector and human scorer measured during sleep stage 2 using kappa coefficient was 0.45, whereas mean agreement between our detector and SIESTA algorithm was 0.62. Our algorithm was also applied to sleep-related memory consolidation data previously analyzed with a SIESTA detector and confirmed previous findings of significant correlation between spindle density and declarative memory consolidation. We then applied our method to a study in monozygotic (MZ) and dizygotic (DZ) twins, examining the genetic component of slow and fast sleep spindle parameters. Our analysis revealed strong genetic influence on variance of all slow spindle parameters, weaker genetic effect on fast spindles, and no effects on fast spindle density and number during stage 2 sleep.

Sleep spindle and K-complex detection using tunable Q-factor wavelet transform and morphological component analysis

A novel framework for joint detection of sleep spindles and K-complex events, two hallmarks of sleep stage S2, is proposed. Sleep electroencephalography (EEG) signals are split into oscillatory (spindles) and transient (K-complex) components. This decomposition is conveniently achieved by applying morphological component analysis (MCA) to a sparse representation of EEG segments obtained by the recently introduced discrete tunable Q-factor wavelet transform (TQWT). Tuning the Q-factor provides a convenient and elegant tool to naturally decompose the signal into an oscillatory and a transient component. The actual detection step relies on thresholding (i) the transient component to reveal K-complexes and (ii) the time-frequency representation of the oscillatory component to identify sleep spindles. Optimal thresholds are derived from ROC-like curves (sensitivity vs. FDR) on training sets and the performance of the method is assessed on test data sets. We assessed the performance of our method using full-night sleep EEG data we collected from 14 participants. In comparison to visual scoring (Expert 1), the proposed method detected spindles with a sensitivity of 83.18% and false discovery rate (FDR) of 39%, while K-complexes were detected with a sensitivity of 81.57% and an FDR of 29.54%. Similar performances were obtained when using a second expert as benchmark. In addition, when the TQWT and MCA steps were excluded from the pipeline the detection sensitivities dropped down to 70% for spindles and to 76.97% for K-complexes, while the FDR rose up to 43.62 and 49.09%, respectively. Finally, we also evaluated the performance of the proposed method on a set of publicly available sleep EEG recordings. Overall, the results we obtained suggest that the TQWT-MCA method may be a valuable alternative to existing spindle and K-complex detection methods. Paths for improvements and further validations with large-scale standard open-access benchmarking data sets are discussed.

Automatic sleep spindle detection: benchmarking with fine temporal resolution using open science tools

Sleep spindle properties index cognitive faculties such as memory consolidation and diseases such as major depression. For this reason, scoring sleep spindle properties in polysomnographic recordings has become an important activity in both research and clinical settings. The tediousness of this manual task has motivated efforts for its automation. Although some progress has been made, increasing the temporal accuracy of spindle scoring and improving the performance assessment methodology are two aspects needing more attention. In this paper, four open-access automated spindle detectors with fine temporal resolution are proposed and tested against expert scoring of two proprietary and two open-access databases. Results highlight several findings: (1) that expert scoring and polysomnographic databases are important confounders when comparing the performance of spindle detectors tested using different databases or scorings; (2) because spindles are sparse events, specificity estimates are potentially misleading for assessing automated detector performance; (3) reporting the performance of spindle detectors exclusively with sensitivity and specificity estimates, as is often seen in the literature, is insufficient; including sensitivity, precision and a more comprehensive statistic such as Matthew's correlation coefficient, F1-score, or Cohen's κ is necessary for adequate evaluation; (4) reporting statistics for some reasonable range of decision thresholds provides a much more complete and useful benchmarking; (5) performance differences between tested automated detectors were found to be similar to those between available expert scorings; (6) much more development is needed to effectively compare the performance of spindle detectors developed by different research teams. Finally, this work clarifies a long-standing but only seldomly posed question regarding whether expert scoring truly is a reliable gold standard for sleep spindle assessment.

Using a quadratic parameter sinusoid model to characterize the structure of EEG sleep spindles

Sleep spindles are essentially non-stationary signals that display time and frequency-varying characteristics within their envelope, which makes it difficult to accurately identify its instantaneous frequency and amplitude. To allow a better parameterization of the structure of spindle, we propose modeling spindles using a Quadratic Parameter Sinusoid (QPS). The QPS is well suited to model spindle activity as it utilizes a quadratic representation to capture the inherent duration and frequency variations within spindles. The effectiveness of our proposed model and estimation technique was quantitatively evaluated in parameter determination experiments using simulated spindle-like signals and real spindles in the presence of background EEG. We used the QPS parameters to predict the energy and frequency of spindles with a mean accuracy of 92.34 and 97.73% respectively. We also show that the QPS parameters provide a quantification of the amplitude and frequency variations occurring within sleep spindles that can be observed visually and related to their characteristic "waxing and waning" shape. We analyze the variations in the parameters values to present how they can be used to understand the inter- and intra-participant variations in spindle structure. Finally, we present a comparison of the QPS parameters of spindles and non-spindles, which shows a substantial difference in parameter values between the two classes.

Sleep spindle alterations in patients with Parkinson's disease

The aim of this study was to identify changes of sleep spindles (SS) in the EEG of patients with Parkinson's disease (PD). Five sleep experts manually identified SS at a central scalp location (C3-A2) in 15 PD and 15 age- and sex-matched control subjects. Each SS was given a confidence score, and by using a group consensus rule, 901 SS were identified and characterized by their (1) duration, (2) oscillation frequency, (3) maximum peak-to-peak amplitude, (4) percent-to-peak amplitude, and (5) density. Between-group comparisons were made for all SS characteristics computed, and significant changes for PD patients vs. control subjects were found for duration, oscillation frequency, maximum peak-to-peak amplitude and density. Specifically, SS density was lower, duration was longer, oscillation frequency slower and maximum peak-to-peak amplitude higher in patients vs. controls. We also computed inter-expert reliability in SS scoring and found a significantly lower reliability in scoring definite SS in patients when compared to controls. How neurodegeneration in PD could influence SS characteristics is discussed. We also note that the SS morphological changes observed here may affect automatic detection of SS in patients with PD or other neurodegenerative disorders (NDDs).

Combining time-frequency and spatial information for the detection of sleep spindles

EEG sleep spindles are short (0.5-2.0 s) bursts of activity in the 11-16 Hz band occurring during non-rapid eye movement (NREM) sleep. This sporadic activity is thought to play a role in memory consolidation, brain plasticity, and protection of sleep integrity. Many automatic detectors have been proposed to assist or replace experts for sleep spindle scoring. However, these algorithms usually detect too many events making it difficult to achieve a good tradeoff between sensitivity (Se) and false detection rate (FDr). In this work, we propose a semi-automatic detector comprising a sensitivity phase based on well-established criteria followed by a specificity phase using spatial and spectral criteria. In the sensitivity phase, selected events are those which amplitude in the 10-16 Hz band and spectral ratio characteristics both reject a null hypothesis (p < 0.1) stating that the considered event is not a spindle. This null hypothesis is constructed from events occurring during rapid eye movement (REM) sleep epochs. In the specificity phase, a hierarchical clustering of the selected candidates is done based on events' frequency and spatial position along the anterior-posterior axis. Only events from the classes grouping most (at least 80%) spindles scored by an expert are kept. We obtain Se = 93.2% and FDr = 93.0% in the first phase and Se = 85.4% and FDr = 86.2% in the second phase. For these two phases, Matthew's correlation coefficients are respectively 0.228 and 0.324. Results suggest that spindles are defined by specific spatio-spectral properties and that automatic detection methods can be improved by considering these features.