A Unified Model of Performance: Validation of its Predictions across Different Sleep/Wake Schedules

Can we rely on wearable sleep-tracker devices for fatigue management?

STUDY OBJECTIVES

Wearable sleep-tracker devices are ubiquitously used to measure sleep; however, the estimated sleep parameters often differ from the gold-standard polysomnography (PSG). It is unclear to what extent we can tolerate these errors within the context of a particular clinical or operational application. Here, we sought to develop a method to quantitatively determine whether a sleep tracker yields acceptable sleep-parameter estimates for assessing alertness impairment.

METHODS

Using literature data, we characterized sleep-measurement errors of 18 unique sleep-tracker devices with respect to PSG. Then, using predictions based on the unified model of performance, we compared the temporal variation of alertness in terms of the psychomotor vigilance test mean response time for simulations with and without added PSG-device sleep-measurement errors, for nominal schedules of 5, 8, or 9 hours of sleep/night or an irregular sleep schedule each night for 30 consecutive days. Finally, we deemed a device error acceptable when the predicted differences were smaller than the within-subject variability of 30 milliseconds. We also established the capability to estimate the extent to which a specific sleep-tracker device meets this acceptance criterion.

RESULTS

On average, the 18 sleep-tracker devices overestimated sleep duration by 19 (standard deviation = 44) minutes. Using these errors for 30 consecutive days, we found that, regardless of sleep schedule, in nearly 80% of the time the resulting predicted alertness differences were smaller than 30 milliseconds.

CONCLUSIONS

We provide a method to quantitatively determine whether a sleep-tracker device produces sleep measurements that are operationally acceptable for fatigue management.

Estimating vigilance from the pre-work shift sleep using an under-mattress sleep sensor

Predicting vigilance impairment in high-risk shift work occupations is critical to help to reduce workplace errors and accidents. Current methods rely on multi-night, often manually entered, sleep data. This study developed a machine learning model for predicting vigilance errors based on a single prior sleep period, derived from an under-mattress sensor. Twenty-four healthy volunteers (mean [SD] age = 27.6 [9.5] years, 12 male) attended the laboratory on two separate occasions, 1 month apart, to compare wake performance and sleep under two different lighting conditions. Each condition occurred over an 8 day protocol comprising a baseline sleep opportunity from 10 p.m. to 7 a.m., a 27 h wake period, then daytime sleep opportunities from 10 a.m. to 7 p.m. on days 3-7. From 12 a.m. to 8 a.m. on each of days 4-7, participants completed simulated night shifts that included six 10 min psychomotor vigilance task (PVT) trials per shift. Sleep was assessed using an under-mattress sensor. Using extra-trees machine learning models, PVT performance (reaction times <500 ms, reaction, and lapses) during each night shift was predicted based on the preceding daytime sleep. The final extra-trees model demonstrated moderate accuracy for predicting PVT performance, with standard errors (RMSE) of 19.9 ms (reaction time, 359 [41.6]ms), 0.42 reactions/s (reaction speed, 2.5 [0.6] reactions/s), and 7.2 (lapses, 10.5 [12.3]). The model also correctly classified 84% of trials containing ≥5 lapses (Matthews correlation coefficient = 0.59, F1 = 0.83). Model performance is comparable to current fatigue prediction models that rely upon self-report or manually entered data. This efficient approach may help to manage fatigue and safety in non-standard work schedules.

A real-time, personalized sleep intervention using mathematical modeling and wearable devices

The prevalence of artificial light exposure has enabled us to be active any time of the day or night, leading to the need for high alertness outside of traditional daytime hours. To address this need, we developed a personalized sleep intervention framework that analyzes real-world sleep-wake patterns obtained from wearable devices to maximize alertness during specific target periods. Our framework utilizes a mathematical model that tracks the dynamic sleep pressure and circadian rhythm based on the user's sleep history. In this way, the model accurately predicts real-time alertness, even for shift workers with complex sleep and work schedules (N = 71, t = 13~21 days). This allowed us to discover a new sleep-wake pattern called the adaptive circadian split sleep, which incorporates a main sleep period and a late nap to enable high alertness during both work and non-work periods of shift workers. We further developed a mobile application that integrates this framework to recommend practical, personalized sleep schedules for individual users to maximize their alertness during a targeted activity time based on their desired sleep onset and available sleep duration. This can reduce the risk of errors for those who require high alertness during nontraditional activity times and improve the health and quality of life for those leading shift work-like lifestyles.

Social dimensions impact individual sleep quantity and quality

While sleep positively impacts well-being, health, and productivity, the effects of societal factors on sleep remain underexplored. Here we analyze the sleep of 30,082 individuals across 11 countries using 52 million activity records from wearable devices. Our data are consistent with past studies of gender and age-associated sleep characteristics. However, our analysis of wearable device data uncovers differences in recorded vs. self-reported bedtime and sleep duration. The dataset allowed us to study how country-specific metrics such as GDP and cultural indices relate to sleep in groups and individuals. Our analysis indicates that diverse sleep metrics can be represented by two dimensions: sleep quantity and quality. We find that 55% of the variation in sleep quality, and 63% in sleep quantity, are explained by societal factors. Within a societal boundary, individual sleep experience was modified by factors like exercise. Increased exercise or daily steps were associated with better sleep quality (for example, faster sleep onset and less time awake in bed), especially in countries like the U.S. and Finland. Understanding how social norms relate to sleep will help create strategies and policies that enhance the positive impacts of sleep on health, such as productivity and well-being.

Assessment of the unified model of performance: accuracy of group-average and individualised alertness predictions

To be effective as a key component of fatigue-management systems, biomathematical models that predict alertness impairment as a function of time of day, sleep history, and caffeine consumption must demonstrate the ability to make accurate predictions across a range of sleep-loss and caffeine schedules. Here, we assessed the ability of the previously reported unified model of performance (UMP) to predict alertness impairment at the group-average and individualised levels in a comprehensive set of 12 studies, including 22 sleep and caffeine conditions, for a total of 301 unique subjects. Given sleep and caffeine schedules, the UMP predicted alertness impairment based on the psychomotor vigilance test (PVT) for the duration of the schedule. To quantify prediction performance, we computed the root mean square error (RMSE) between model predictions and PVT data, and the fraction of measured PVTs that fell within the models' prediction intervals (PIs). For the group-average model predictions, the overall RMSE was 43 ms (range 15-74 ms) and the fraction of PVTs within the PIs was 80% (range 41%-100%). At the individualised level, the UMP could predict alertness for 81% of the subjects, with an overall average RMSE of 64 ms (range 32-147 ms) and fraction of PVTs within the PIs conservatively estimated as 71% (range 41%-100%). Altogether, these results suggest that, for the group-average model and 81% of the individualised models, in three out of four PVT measurements we cannot distinguish between study data and model predictions.

2B-Alert Web 2.0, an Open-Access Tool for Predicting Alertness and Optimizing the Benefits of Caffeine: Utility Study

BACKGROUND

One-third of the US population experiences sleep loss, with the potential to impair physical and cognitive performance, reduce productivity, and imperil safety during work and daily activities. Computer-based fatigue-management systems with the ability to predict the effects of sleep schedules on alertness and identify safe and effective caffeine interventions that maximize its stimulating benefits could help mitigate cognitive impairment due to limited sleep. To provide these capabilities to broad communities, we previously released 2B-Alert Web, a publicly available tool for predicting the average alertness level of a group of individuals as a function of time of day, sleep history, and caffeine consumption.

OBJECTIVE

In this study, we aim to enhance the capability of the 2B-Alert Web tool by providing the means for it to automatically recommend safe and effective caffeine interventions (time and dose) that lead to optimal alertness levels at user-specified times under any sleep-loss condition.

METHODS

We incorporated a recently developed caffeine-optimization algorithm into the predictive models of the original 2B-Alert Web tool, allowing the system to search for and identify viable caffeine interventions that result in user-specified alertness levels at desired times of the day. To assess the potential benefits of this new capability, we simulated four sleep-deprivation conditions (sustained operations, restricted sleep with morning or evening shift, and night shift with daytime sleep) and compared the alertness levels resulting from the algorithm's recommendations with those based on the US Army caffeine-countermeasure guidelines. In addition, we enhanced the usability of the tool by adopting a drag-and-drop graphical interface for the creation of sleep and caffeine schedules.

RESULTS

For the 4 simulated conditions, the 2B-Alert Web-proposed interventions increased mean alertness by 36% to 94% and decreased peak alertness impairment by 31% to 71% while using equivalent or smaller doses of caffeine as the corresponding US Army guidelines.

CONCLUSIONS

The enhanced capability of this evidence-based, publicly available tool increases the efficiency by which diverse communities of users can identify safe and effective caffeine interventions to mitigate the effects of sleep loss in the design of research studies and work and rest schedules.

Optimal Sleep and Work Schedules to Maximize Alertness

STUDY OBJECTIVES

Working outside the conventional "9-to-5" shift may lead to reduced sleep and alertness impairment. Here, we developed an optimization algorithm to identify sleep and work schedules that minimize alertness impairment during work hours, while reducing impairment during non-work hours.

METHODS

The optimization algorithm searches among a large number of possible sleep and work schedules and estimates their effectiveness in mitigating alertness impairment using the Unified Model of Performance (UMP). To this end, the UMP, and its extensions to estimate sleep latency and sleep duration, predicts the time course of alertness of each potential schedule and their physiological feasibility. We assessed the algorithm by simulating four experimental studies, where we compared alertness levels during work periods for sleep schedules proposed by the algorithm against those used in the studies. In addition, in one of the studies we assessed the algorithm's ability to simultaneously optimize sleep and work schedules.

RESULTS

Using the same amount of sleep as in the studies but distributing it optimally, the sleep schedules proposed by the optimization algorithm reduced alertness impairment during work periods by an average of 29%. Similarly, simultaneously optimized sleep and work schedules, for a recovery period following a chronic sleep restriction challenge, accelerated the return to baseline levels by two days when compared to the conventional 9-to-5 work schedule.

CONCLUSIONS

Our work provides the first quantitative tool to optimize sleep and work schedules and extends the capabilities of existing fatigue-management tools.

Models for predicting sleep latency and sleep duration

STUDY OBJECTIVES

Planning effective sleep-wake schedules for civilian and military settings depends on the ability to predict the extent to which restorative sleep is likely for a specified sleep period. Here, we developed and validated two mathematical models, one for predicting sleep latency and a second for predicting sleep duration, as decision aids to predict efficacious sleep periods.

METHODS

We extended the Unified Model of Performance (UMP), a well-validated mathematical model of neurobehavioral performance, to predict sleep latency and sleep duration, which vary nonlinearly as a function of the homeostatic sleep pressure and the circadian rhythm. To this end, we used the UMP to predict the time course of neurobehavioral performance under different conditions. We developed and validated the models using experimental data from 317 unique subjects from 24 different studies, which included sleep conditions spanning the entire circadian cycle.

RESULTS

The sleep-latency and sleep-duration models accounted for 42% and 84% of the variance in the data, respectively, and yielded acceptable average prediction errors for planning sleep schedules (4.0 min for sleep latency and 0.8 h for sleep duration). Importantly, we identified conditions under which small shifts in sleep onset timing result in disproportionately large differences in sleep duration-knowledge that may be applied to improve performance, safety, and sustainability in civilian and military operations.

CONCLUSIONS

These models extend the capabilities of existing predictive fatigue-management tools, allowing users to anticipate the most opportune times to schedule sleep periods.

The effects of seasons and weather on sleep patterns measured through longitudinal multimodal sensing

Previous studies of seasonal effects on sleep have yielded unclear results, likely due to methodological differences and limitations in data size and/or quality. We measured the sleep habits of 216 individuals across the U.S. over four seasons for slightly over a year using objective, continuous, and unobtrusive measures of sleep and local weather. In addition, we controlled for demographics and trait-like constructs previously identified to correlate with sleep behavior. We investigated seasonal and weather effects of sleep duration, bedtime, and wake time. We found several small but statistically significant effects of seasonal and weather effects on sleep patterns. We observe the strongest seasonal effects for wake time and sleep duration, especially during the spring season: wake times are earlier, and sleep duration decreases (compared to the reference season winter). Sleep duration also modestly decreases when day lengths get longer (between the winter and summer solstice). Bedtimes and wake times tend to be slightly later as outdoor temperature increases.

Recent advances in modeling sleep: from the clinic to society and disease

In the past few decades, advances in understanding sleep-wake neurophysiology have occurred hand-in-hand with advances in mathematical modeling of sleep and wake. In this review, we summarize recent updates in modeling the timing and durations of sleep and wake, the underlying neurophysiology of sleep and wake, and the application of these models in understanding cognition and disease. Throughout, we highlight the role modeling has played in developing our understanding of sleep and its underlying mechanisms. We present open questions and controversies in the field and propose the utility of individualized models of sleep for precision sleep medicine.

Differential effects of split and continuous sleep on neurobehavioral function and glucose tolerance in sleep-restricted adolescents

Study Objectives

Many adolescents are exposed to sleep restriction on school nights. We assessed how different apportionment of restricted sleep (continuous vs. split sleep) influences neurobehavioral function and glucose levels.

Methods

Adolescents, aged 15–19 years, were evaluated in a dormitory setting using a parallel-group design. Following two baseline nights of 9-hour time-in-bed (TIB), participants underwent either 5 nights of continuous 6.5-h TIB (n = 29) or 5-hour nocturnal TIB with a 1.5-hour afternoon nap (n = 29). After two recovery nights of 9-hour TIB, participants were sleep restricted for another three nights. Sleep was assessed using polysomnography (PSG). Cognitive performance and mood were evaluated three times per day. Oral glucose tolerance tests (OGTT) were conducted on mornings after baseline sleep, recovery sleep, and the third day of each sleep restriction cycle.

Results

The split sleep group had fewer vigilance lapses, better working memory and executive function, faster processing speed, lower level of subjective sleepiness, and more positive mood, even though PSG-verified total sleep time was less than the continuous sleep group. However, vigilance in both sleep-restricted groups was inferior to adolescents in a prior sample given 9-hour nocturnal TIB. During both cycles of sleep restriction, blood glucose during the OGTT increased by a greater amount in the split sleep schedule compared with persons receiving 6.5-hour continuous sleep.

Conclusions

In adolescents, modest multinight sleep restriction had divergent negative effects on cognitive performance and glucose levels depending on how the restricted sleep was apportioned. They are best advised to obtain the recommended amount of nocturnal sleep.

Trial registration

https://clinicaltrials.gov/ct2/show/NCT03333512

Sleep–wake behaviors exhibited by shift workers in normal operations and predicted by a biomathematical model of fatigue

Study Objectives

To compare rail workers’ actual sleep–wake behaviors in normal operations to those predicted by a biomathematical model of fatigue (BMMF). To determine whether there are group-level residual sources of error in sleep predictions that could be modeled to improve group-level sleep predictions.

Methods

The sleep–wake behaviors of 354 rail workers were examined during 1,722 breaks that were 8–24 h in duration. Sleep–wake patterns were continuously monitored using wrist-actigraphy and predicted from the work–rest schedule using a BMMF. Rail workers’ actual and predicted sleep–wake behaviors were defined as split-sleep (i.e. ≥2 sleep periods in a break) and consolidated-sleep (i.e. one sleep period in a break) behaviors. Sleepiness was predicted from the actual and predicted sleep–wake data.

Results

Consolidated-sleep behaviors were observed during 1,441 breaks and correctly predicted during 1,359 breaks. Split-sleep behaviors were observed during 280 breaks and correctly predicted during 182 breaks. Predicting the wrong type of sleep–wake behavior resulted in a misestimation of hours of sleep during a break. Relative to sleepiness predictions derived from actual sleep–wake data, predicting the wrong type of sleep–wake behavior resulted in a misestimation of sleepiness predictions during the subsequent shift.

Conclusions

All workers with the same work–rest schedule have the same predicted sleep–wake behaviors; however, these workers do not all exhibit the same sleep–wake behaviors in real-world operations. Future models could account for this group-level residual variance with a new approach to modeling sleep, whereby sub-group(s) may be predicted to exhibit one of a number of sleep–wake behaviors.

Sleep Modelling across Physiological Levels

Sleep and circadian rhythms are regulated across multiple functional, spatial and temporal levels: from genes to networks of coupled neurons and glial cells, to large scale brain dynamics and behaviour. The dynamics at each of these levels are complex and the interaction between the levels is even more so, so research have mostly focused on interactions within the levels to understand the underlying mechanisms—the so-called reductionist approach. Mathematical models were developed to test theories of sleep regulation and guide new experiments at each of these levels and have become an integral part of the field. The advantage of modelling, however, is that it allows us to simulate and test the dynamics of complex biological systems and thus provides a tool to investigate the connections between the different levels and study the system as a whole. In this paper I review key models of sleep developed at different physiological levels and discuss the potential for an integrated systems biology approach for sleep regulation across these levels. I also highlight the necessity of building mechanistic connections between models of sleep and circadian rhythms across these levels.

2B-Alert App: A mobile application for real-time individualized prediction of alertness

Knowing how an individual responds to sleep deprivation is a requirement for developing personalized fatigue management strategies. Here we describe and validate the 2B‐Alert App, the first mobile application that progressively learns an individual’s trait‐like response to sleep deprivation in real time, to generate increasingly more accurate individualized predictions of alertness. We incorporated a Bayesian learning algorithm within the validated Unified Model of Performance to automatically and gradually adapt the model parameters to an individual after each psychomotor vigilance test. We implemented the resulting model and the psychomotor vigilance test as a smartphone application (2B‐Alert App), and prospectively validated its performance in a 62‐hr total sleep deprivation study in which 21 participants used the app to perform psychomotor vigilance tests every 3 hr and obtain real‐time individualized predictions after each test. The temporal profiles of reaction times on the app‐conducted psychomotor vigilance tests were well correlated with and as sensitive as those obtained with a previously characterized psychomotor vigilance test device. The app progressively learned each individual’s trait‐like response to sleep deprivation throughout the study, yielding increasingly more accurate predictions of alertness for the last 24 hr of total sleep deprivation as the number of psychomotor vigilance tests increased. After only 12 psychomotor vigilance tests, the accuracy of the model predictions was comparable to the peak accuracy obtained using all psychomotor vigilance tests. With the ability to make real‐time individualized predictions of the effects of sleep deprivation on future alertness, the 2B‐Alert App can be used to tailor personalized fatigue management strategies, facilitating self‐management of alertness and safety in operational and non‐operational settings.

Improved Mental Acuity Forecasting with an Individualized Quantitative Sleep Model

Sleep impairment significantly alters human brain structure and cognitive function, but available evidence suggests that adults in developed nations are sleeping less. A growing body of research has sought to use sleep to forecast cognitive performance by modeling the relationship between the two, but has generally focused on vigilance rather than other cognitive constructs affected by sleep, such as reaction time, executive function, and working memory. Previous modeling efforts have also utilized subjective, self-reported sleep durations and were restricted to laboratory environments. In the current effort, we addressed these limitations by employing wearable systems and mobile applications to gather objective sleep information, assess multi-construct cognitive performance, and model/predict changes to mental acuity. Thirty participants were recruited for participation in the study, which lasted 1 week. Using the Fitbit Charge HR and a mobile version of the automated neuropsychological assessment metric called CogGauge, we gathered a series of features and utilized the unified model of performance to predict mental acuity based on sleep records. Our results suggest that individuals poorly rate their sleep duration, supporting the need for objective sleep metrics to model circadian changes to mental acuity. Participant compliance in using the wearable throughout the week and responding to the CogGauge assessments was 80%. Specific biases were identified in temporal metrics across mobile devices and operating systems and were excluded from the mental acuity metric development. Individualized prediction of mental acuity consistently outperformed group modeling. This effort indicates the feasibility of creating an individualized, mobile assessment and prediction of mental acuity, compatible with the majority of current mobile devices.

2B-Alert Web: An Open-Access Tool for Predicting the Effects of Sleep/Wake Schedules and Caffeine Consumption on Neurobehavioral Performance

STUDY OBJECTIVES

Computational tools that predict the effects of daily sleep/wake amounts on neurobehavioral performance are critical components of fatigue management systems, allowing for the identification of periods during which individuals are at increased risk for performance errors. However, none of the existing computational tools is publicly available, and the commercially available tools do not account for the beneficial effects of caffeine on performance, limiting their practical utility. Here, we introduce 2B-Alert Web, an open-access tool for predicting neurobehavioral performance, which accounts for the effects of sleep/wake schedules, time of day, and caffeine consumption, while incorporating the latest scientific findings in sleep restriction, sleep extension, and recovery sleep.

METHODS

We combined our validated Unified Model of Performance and our validated caffeine model to form a single, integrated modeling framework instantiated as a Web-enabled tool. 2B-Alert Web allows users to input daily sleep/wake schedules and caffeine consumption (dosage and time) to obtain group-average predictions of neurobehavioral performance based on psychomotor vigilance tasks. 2B-Alert Web is accessible at: https://2b-alert-web.bhsai.org.

RESULTS

The 2B-Alert Web tool allows users to obtain predictions for mean response time, mean reciprocal response time, and number of lapses. The graphing tool allows for simultaneous display of up to seven different sleep/wake and caffeine schedules. The schedules and corresponding predicted outputs can be saved as a Microsoft Excel file; the corresponding plots can be saved as an image file. The schedules and predictions are erased when the user logs off, thereby maintaining privacy and confidentiality.

CONCLUSIONS

The publicly accessible 2B-Alert Web tool is available for operators, schedulers, and neurobehavioral scientists as well as the general public to determine the impact of any given sleep/wake schedule, caffeine consumption, and time of day on performance of a group of individuals. This evidence-based tool can be used as a decision aid to design effective work schedules, guide the design of future sleep restriction and caffeine studies, and increase public awareness of the effects of sleep amounts, time of day, and caffeine on alertness.

A Unified Model of Performance for Predicting the Effects of Sleep and Caffeine

STUDY OBJECTIVES

Existing mathematical models of neurobehavioral performance cannot predict the beneficial effects of caffeine across the spectrum of sleep loss conditions, limiting their practical utility. Here, we closed this research gap by integrating a model of caffeine effects with the recently validated unified model of performance (UMP) into a single, unified modeling framework. We then assessed the accuracy of this new UMP in predicting performance across multiple studies.

METHODS

We hypothesized that the pharmacodynamics of caffeine vary similarly during both wakefulness and sleep, and that caffeine has a multiplicative effect on performance. Accordingly, to represent the effects of caffeine in the UMP, we multiplied a dose-dependent caffeine factor (which accounts for the pharmacokinetics and pharmacodynamics of caffeine) to the performance estimated in the absence of caffeine. We assessed the UMP predictions in 14 distinct laboratory- and field-study conditions, including 7 different sleep-loss schedules (from 5 h of sleep per night to continuous sleep loss for 85 h) and 6 different caffeine doses (from placebo to repeated 200 mg doses to a single dose of 600 mg).

RESULTS

The UMP accurately predicted group-average psychomotor vigilance task performance data across the different sleep loss and caffeine conditions (6% < error < 27%), yielding greater accuracy for mild and moderate sleep loss conditions than for more severe cases. Overall, accounting for the effects of caffeine resulted in improved predictions (after caffeine consumption) by up to 70%.

CONCLUSIONS

The UMP provides the first comprehensive tool for accurate selection of combinations of sleep schedules and caffeine countermeasure strategies to optimize neurobehavioral performance.