Validation Framework for Sleep Stage Scoring in Wearable Sleep Trackers and Monitors with Polysomnography Ground Truth

REM sleep breathing: Insights beyond conventional respiratory metrics

Breathing and sleep state are tightly linked. The traditional approach to evaluation of breathing in rapid eye movement sleep has been to focus on apneas and hypopneas, and associated hypoxia or hypercapnia. However, rapid eye movement sleep breathing offers novel insights into sleep physiology and pathology, secondary to complex interactions of rapid eye movement state and cardiorespiratory biology. In this review, morphological analysis of clinical polysomnogram data to assess respiratory patterns and associations across a range of health and disease is presented. There are several relatively unique insights that may be evident by assessment of breathing during rapid eye movement sleep. These include the original discovery of rapid eye movement sleep and scoring of neonatal sleep, control of breathing in rapid eye movement sleep, rapid eye movement sleep homeostasis, sleep apnea endotyping and pharmacotherapy, rapid eye movement sleep stability, non-electroencephalogram sleep staging, influences on cataplexy, mimics of rapid eye movement behaviour disorder, a reflection of autonomic health, and insights into cardiac arrhythmogenesis. In summary, there is rich clinically actionable information beyond sleep apnea encoded in the respiratory patterns of rapid eye movement sleep.

Metrology of two wearable sleep trackers against polysomnography in patients with sleep complaints

Sleep trackers are used widely by patients with sleep complaints, however their metrological validation is often poor and relies on healthy subjects. We assessed the metrological validity of two commercially available sleep trackers (Withings Activité/Fitbit Alta HR) through a prospective observational monocentric study, in adult patients referred for polysomnography (PSG). We compared the total sleep time (TST), REM time, REM latency, nonREM1 + 2 time, nonREM3 time, and wake after sleep onset (WASO). We report absolute and relative errors, Bland-Altman representations, and a contingency table of times spent in sleep stages with respect to PSG. Sixty-five patients were included (final sample size 58 for Withings and 52 for Fitbit). Both devices gave a relatively accurate sleep start time with a median absolute error of 5 (IQR -43; 27) min for Withings and -2.0 (-12.5; 4.2) min for Fitbit but both overestimated TST. Withings tended to underestimate WASO with a median absolute error of -25.0 (-61.5; -8.5) min, while Fitbit tended to overestimate it (median absolute error 10 (-18; 43) min. Withings underestimated light sleep and overestimated deep sleep, while Fitbit overestimated light and REM sleep and underestimated deep sleep. The overall kappas for concordance of each epoch between PSG and devices were low: 0.12 (95%CI 0.117-0.121) for Withings and VPSG indications 0.07 (95%CI 0.067-0.071) for Fitbit, as well as kappas for each VPSG indication 0.07 (95%CI 0.067-0.071). Thus, commercially available sleep trackers are not reliable for sleep architecture in patients with sleep complaints/pathologies and should not replace actigraphy and/or PSG.

Comparative analysis of sleep parameters and structures derived from wearable flexible electrode sleep patches and polysomnography in young adults

Polysomnography (PSG) is the gold standard for clinical sleep monitoring, but its cost, discomfort, and limited suitability for continuous use present challenges. The flexible electrode sleep patch (FESP) emerges as an economically viable and patient-friendly solution, offering lightweight, simple operation, and self-applicable. Nevertheless, its utilization in young individuals remains uncertain. The objective of this study was to compare sleep data obtained by FESP and PSG in healthy young individuals and analyze agreement for sleep parameters and structure classification. Overnight monitoring with FESP and PSG recordings in 48 participants (mean age: 23 yr) was done. Correlation analysis, Bland-Altman plots, and Cohen's kappa coefficient assessed consistency. Sensitivity, specificity, and predictive values compared classification against PSG. FESP showed strong correlation and consistency with PSG for sleep monitoring. Bland-Altman plots indicated small errors and high consistency. Kappa values (0.70-0.84) suggested substantial agreement for sleep stage classification. Pearson correlation coefficient values for sleep stages (0.75-0.88) and sleep parameters (0.80-0.96) confirm that FESP has a strong application. Intraclass correlation coefficient yielded values between 0.65 and 0.97. In addition, FESP demonstrated an impressive accuracy range of 84.12-93.47% for sleep stage classification. The FESP also features a wearable self-test program with an error rate of no more than 8% for both deep sleep and wake. In young adults, FESP demonstrated reliable monitoring capabilities comparable to PSG. With its low cost and user-friendly design, FESP is a potential alternative for portable sleep assessment in clinical and research applications. Further studies involving larger populations are needed to validate its diagnostic potential.NEW & NOTEWORTHY By comparison with PSG, this study confirmed the reliability of an efficient, objective, low-cost, and noninvasive portable automatic sleep-monitoring device FESP, which provides effective information for long-term family sleep disorder diagnosis and sleep quality monitoring.

Studying sleep: towards the identification of hypnogram features that drive expert interpretation

STUDY OBJECTIVES

Hypnograms contain a wealth of information and play an important role in sleep medicine. However, interpretation of the hypnogram is a difficult task and requires domain knowledge and "clinical intuition." This study aimed to uncover which features of the hypnogram drive interpretation by physicians. In other words, make explicit which features physicians implicitly look for in hypnograms.

METHODS

Three sleep experts evaluated up to 612 hypnograms, indicating normal or abnormal sleep structure and suspicion of disorders. ElasticNet and convolutional neural network classification models were trained to predict the collected expert evaluations using hypnogram features and stages as input. The models were evaluated using several measures, including accuracy, Cohen's kappa, Matthew's correlation coefficient, and confusion matrices. Finally, model coefficients and visual analytics techniques were used to interpret the models to associate hypnogram features with expert evaluation.

RESULTS

Agreement between models and experts (Kappa between 0.47 and 0.52) is similar to agreement between experts (Kappa between 0.38 and 0.50). Sleep fragmentation, measured by transitions between sleep stages per hour, and sleep stage distribution were identified as important predictors for expert interpretation.

CONCLUSIONS

By comparing hypnograms not solely on an epoch-by-epoch basis, but also on these more specific features that are relevant for the evaluation of experts, performance assessment of (automatic) sleep-staging and surrogate sleep trackers may be improved. In particular, sleep fragmentation is a feature that deserves more attention as it is often not included in the PSG report, and existing (wearable) sleep trackers have shown relatively poor performance in this aspect.

Can we rely on wearable sleep-tracker devices for fatigue management?

STUDY OBJECTIVES

Wearable sleep-tracker devices are ubiquitously used to measure sleep; however, the estimated sleep parameters often differ from the gold-standard polysomnography (PSG). It is unclear to what extent we can tolerate these errors within the context of a particular clinical or operational application. Here, we sought to develop a method to quantitatively determine whether a sleep tracker yields acceptable sleep-parameter estimates for assessing alertness impairment.

METHODS

Using literature data, we characterized sleep-measurement errors of 18 unique sleep-tracker devices with respect to PSG. Then, using predictions based on the unified model of performance, we compared the temporal variation of alertness in terms of the psychomotor vigilance test mean response time for simulations with and without added PSG-device sleep-measurement errors, for nominal schedules of 5, 8, or 9 hours of sleep/night or an irregular sleep schedule each night for 30 consecutive days. Finally, we deemed a device error acceptable when the predicted differences were smaller than the within-subject variability of 30 milliseconds. We also established the capability to estimate the extent to which a specific sleep-tracker device meets this acceptance criterion.

RESULTS

On average, the 18 sleep-tracker devices overestimated sleep duration by 19 (standard deviation = 44) minutes. Using these errors for 30 consecutive days, we found that, regardless of sleep schedule, in nearly 80% of the time the resulting predicted alertness differences were smaller than 30 milliseconds.

CONCLUSIONS

We provide a method to quantitatively determine whether a sleep-tracker device produces sleep measurements that are operationally acceptable for fatigue management.

A systematic review of the performance of actigraphy in measuring sleep stages

The accuracy of actigraphy for sleep staging is assumed to be poor, but examination is limited. This systematic review aimed to assess the performance of actigraphy in sleep stage classification of adults. A systematic search was performed using MEDLINE, Web of Science, Google Scholar, and Embase databases. We identified eight studies that compared sleep architecture estimates between wrist-worn actigraphy and polysomnography. Large heterogeneity was found with respect to how sleep stages were grouped, and the choice of metrics used to evaluate performance. Quantitative synthesis was not possible, so we performed a narrative synthesis of the literature. From the limited number of studies, we found that actigraphy-based sleep staging had some ability to classify different sleep stages compared with polysomnography.

In-Home Smartphone-Based Prediction of Obstructive Sleep Apnea in Conjunction With Level 2 Home Polysomnography

Importance

Consumer-level sleep analysis technologies have the potential to revolutionize the screening for obstructive sleep apnea (OSA). However, assessment of OSA prediction models based on in-home recording data is usually performed concurrently with level 1 in-laboratory polysomnography (PSG). Establishing the predictability of OSA using sound data recorded from smartphones based on level 2 PSG at home is important.

Objective

To validate the performance of a prediction model for OSA using breathing sound recorded from smartphones in conjunction with level 2 PSG at home.

Design, Setting, and Participants

This diagnostic study followed a prospective design, involving participants who underwent unattended level 2 home PSG. Breathing sounds were recorded during sleep using 2 smartphones, one with an iOS operating system and the other with an Android operating system, simultaneously with home PSG in participants' own home environment. Participants were 19 years and older, slept alone, and had either been diagnosed with OSA or had no previous diagnosis. The study was performed between February 2022 and February 2023.

Main Outcomes and Measures

Sensitivity, specificity, positive predictive value, negative predictive value, and accuracy of the predictive model based on the recorded breathing sounds.

Results

Of the 101 participants included during the study duration, the mean (SD) age was 48.3 (14.9) years, and 51 (50.5%) were female. For the iOS smartphone, the sensitivity values at apnea-hypopnea index (AHI) levels of 5, 15, and 30 per hour were 92.6%, 90.9%, and 93.3%, respectively, with specificities of 84.3%, 94.4%, and 94.4%, respectively. Similarly, for the Android smartphone, the sensitivity values at AHI levels of 5, 15, and 30 per hour were 92.2%, 90.0%, and 92.9%, respectively, with specificities of 84.0%, 94.4%, and 94.3%, respectively. The accuracy for the iOS smartphone was 88.6%, 93.3%, and 94.3%, respectively, and for the Android smartphone was 88.1%, 93.1%, and 94.1% at AHI levels of 5, 15, and 30 per hour, respectively.

Conclusions and Relevance

This diagnostic study demonstrated the feasibility of predicting OSA with a reasonable level of accuracy using breathing sounds obtained by smartphones during sleep at home.

Clinical utility of smartphone applications for sleep physicians

PURPOSE

To review various smartphone applications (apps) for sleep architecture and screening of obstructive sleep apnea (OSA) and to outline their utility for sleep physicians.

METHODS

Mobile application stores (Google Play and Apple iOS App Store) were searched for sleep analysis applications (apps) that are targeted for consumer use. Apps were identified by two independent investigators for apps published through July 2022. App information including parameters obtained for sleep analysis were extracted from each app.

RESULTS

The search identified 50 apps that reported sufficient outcome measures to be considered for assessment. Half of the apps tracked sleep with phone-only technology, while 19 utilized sleep and fitness trackers, three utilized sleep-only wearable devices, and three utilized nearable devices. Seven apps provided data useful for tracking users for signs and symptoms of obstructive sleep apnea.

CONCLUSION

There are a variety of sleep analysis apps available on the market to consumers currently. Though the sleep analysis of these apps may not be validated, sleep physicians should be aware of these apps to improve understanding and education of their patients.

Accuracy of 11 Wearable, Nearable, and Airable Consumer Sleep Trackers: Prospective Multicenter Validation Study

BACKGROUND

Consumer sleep trackers (CSTs) have gained significant popularity because they enable individuals to conveniently monitor and analyze their sleep. However, limited studies have comprehensively validated the performance of widely used CSTs. Our study therefore investigated popular CSTs based on various biosignals and algorithms by assessing the agreement with polysomnography.

OBJECTIVE

This study aimed to validate the accuracy of various types of CSTs through a comparison with in-lab polysomnography. Additionally, by including widely used CSTs and conducting a multicenter study with a large sample size, this study seeks to provide comprehensive insights into the performance and applicability of these CSTs for sleep monitoring in a hospital environment.

METHODS

The study analyzed 11 commercially available CSTs, including 5 wearables (Google Pixel Watch, Galaxy Watch 5, Fitbit Sense 2, Apple Watch 8, and Oura Ring 3), 3 nearables (Withings Sleep Tracking Mat, Google Nest Hub 2, and Amazon Halo Rise), and 3 airables (SleepRoutine, SleepScore, and Pillow). The 11 CSTs were divided into 2 groups, ensuring maximum inclusion while avoiding interference between the CSTs within each group. Each group (comprising 8 CSTs) was also compared via polysomnography.

RESULTS

The study enrolled 75 participants from a tertiary hospital and a primary sleep-specialized clinic in Korea. Across the 2 centers, we collected a total of 3890 hours of sleep sessions based on 11 CSTs, along with 543 hours of polysomnography recordings. Each CST sleep recording covered an average of 353 hours. We analyzed a total of 349,114 epochs from the 11 CSTs compared with polysomnography, where epoch-by-epoch agreement in sleep stage classification showed substantial performance variation. More specifically, the highest macro F1 score was 0.69, while the lowest macro F1 score was 0.26. Various sleep trackers exhibited diverse performances across sleep stages, with SleepRoutine excelling in the wake and rapid eye movement stages, and wearables like Google Pixel Watch and Fitbit Sense 2 showing superiority in the deep stage. There was a distinct trend in sleep measure estimation according to the type of device. Wearables showed high proportional bias in sleep efficiency, while nearables exhibited high proportional bias in sleep latency. Subgroup analyses of sleep trackers revealed variations in macro F1 scores based on factors, such as BMI, sleep efficiency, and apnea-hypopnea index, while the differences between male and female subgroups were minimal.

CONCLUSIONS

Our study showed that among the 11 CSTs examined, specific CSTs showed substantial agreement with polysomnography, indicating their potential application in sleep monitoring, while other CSTs were partially consistent with polysomnography. This study offers insights into the strengths of CSTs within the 3 different classes for individuals interested in wellness who wish to understand and proactively manage their own sleep.

Performance of a multisensor smart ring to evaluate sleep: in-lab and home-based evaluation of generalized and personalized algorithms

STUDY OBJECTIVES

Wearable sleep technology has rapidly expanded across the consumer market due to advances in technology and increased interest in personalized sleep assessment to improve health and mental performance. We tested the performance of a novel device, the Happy Ring, alongside other commercial wearables (Actiwatch 2, Fitbit Charge 4, Whoop 3.0, Oura Ring V2), against in-lab polysomnography (PSG) and at-home electroencephalography (EEG)-derived sleep monitoring device, the Dreem 2 Headband.

METHODS

Thirty-six healthy adults with no diagnosed sleep disorders and no recent use of medications or substances known to affect sleep patterns were assessed across 77 nights. Subjects participated in a single night of in-lab PSG and two nights of at-home data collection. The Happy Ring includes sensors for skin conductance, movement, heart rate, and skin temperature. The Happy Ring utilized two machine-learning derived scoring algorithms: a "generalized" algorithm that applied broadly to all users, and a "personalized" algorithm that adapted to individual subjects' data. Epoch-by-epoch analyses compared the wearable devices to in-lab PSG and to at-home EEG Headband.

RESULTS

Compared to in-lab PSG, the "generalized" and "personalized" algorithms demonstrated good sensitivity (94% and 93%, respectively) and specificity (70% and 83%, respectively). The Happy Personalized model demonstrated a lower bias and more narrow limits of agreement across Bland-Altman measures.

CONCLUSION

The Happy Ring performed well at home and in the lab, especially regarding sleep/wake detection. The personalized algorithm demonstrated improved detection accuracy over the generalized approach and other devices, suggesting that adaptable, dynamic algorithms can enhance sleep detection accuracy.