Detecting the impact of subject characteristics on machine learning-based diagnostic applications

Transformer-based transfer learning on self-reported voice recordings for Parkinson's disease diagnosis

Deep learning (DL) techniques are becoming more popular for diagnosing Parkinson's disease (PD) because they offer non-invasive and easily accessible tools. By using advanced data analysis, these methods improve early detection and diagnosis, which is crucial for managing the disease effectively. This study explores end-to-end DL architectures, such as convolutional neural networks and transformers, for diagnosing PD using self-reported voice data collected via smartphones in everyday settings. Transfer learning was applied by starting with models pre-trained on large datasets from the image and the audio domains and then fine-tuning them on the mPower voice data. The Transformer model pre-trained on the voice data performed the best, achieving an average AUC of [Formula: see text] and an average AUPRC of [Formula: see text], outperforming models trained from scratch. To the best of our knowledge, this is the first use of a Transformer model for audio data in PD diagnosis, using this dataset. We achieved better results than previous studies, whether they focused solely on the voice or incorporated multiple modalities, by relying only on the voice as a biomarker. These results show that using self-reported voice data with state-of-the-art DL architectures can significantly improve PD prediction and diagnosis, potentially leading to better patient outcomes.

Data leakage in deep learning studies of translational EEG

A growing number of studies apply deep neural networks (DNNs) to recordings of human electroencephalography (EEG) to identify a range of disorders. In many studies, EEG recordings are split into segments, and each segment is randomly assigned to the training or test set. As a consequence, data from individual subjects appears in both the training and the test set. Could high test-set accuracy reflect data leakage from subject-specific patterns in the data, rather than patterns that identify a disease? We address this question by testing the performance of DNN classifiers using segment-based holdout (in which segments from one subject can appear in both the training and test set), and comparing this to their performance using subject-based holdout (where all segments from one subject appear exclusively in either the training set or the test set). In two datasets (one classifying Alzheimer's disease, and the other classifying epileptic seizures), we find that performance on previously-unseen subjects is strongly overestimated when models are trained using segment-based holdout. Finally, we survey the literature and find that the majority of translational DNN-EEG studies use segment-based holdout. Most published DNN-EEG studies may dramatically overestimate their classification performance on new subjects.

Harnessing Consumer Wearable Digital Biomarkers for Individualized Recognition of Postpartum Depression Using the All of Us Research Program Data Set: Cross-Sectional Study

BACKGROUND

Postpartum depression (PPD) poses a significant maternal health challenge. The current approach to detecting PPD relies on in-person postpartum visits, which contributes to underdiagnosis. Furthermore, recognizing PPD symptoms can be challenging. Therefore, we explored the potential of using digital biomarkers from consumer wearables for PPD recognition.

OBJECTIVE

The main goal of this study was to showcase the viability of using machine learning (ML) and digital biomarkers related to heart rate, physical activity, and energy expenditure derived from consumer-grade wearables for the recognition of PPD.

METHODS

Using the All of Us Research Program Registered Tier v6 data set, we performed computational phenotyping of women with and without PPD following childbirth. Intraindividual ML models were developed using digital biomarkers from Fitbit to discern between prepregnancy, pregnancy, postpartum without depression, and postpartum with depression (ie, PPD diagnosis) periods. Models were built using generalized linear models, random forest, support vector machine, and k-nearest neighbor algorithms and evaluated using the κ statistic and multiclass area under the receiver operating characteristic curve (mAUC) to determine the algorithm with the best performance. The specificity of our individualized ML approach was confirmed in a cohort of women who gave birth and did not experience PPD. Moreover, we assessed the impact of a previous history of depression on model performance. We determined the variable importance for predicting the PPD period using Shapley additive explanations and confirmed the results using a permutation approach. Finally, we compared our individualized ML methodology against a traditional cohort-based ML model for PPD recognition and compared model performance using sensitivity, specificity, precision, recall, and F1-score.

RESULTS

Patient cohorts of women with valid Fitbit data who gave birth included <20 with PPD and 39 without PPD. Our results demonstrated that intraindividual models using digital biomarkers discerned among prepregnancy, pregnancy, postpartum without depression, and postpartum with depression (ie, PPD diagnosis) periods, with random forest (mAUC=0.85; κ=0.80) models outperforming generalized linear models (mAUC=0.82; κ=0.74), support vector machine (mAUC=0.75; κ=0.72), and k-nearest neighbor (mAUC=0.74; κ=0.62). Model performance decreased in women without PPD, illustrating the method's specificity. Previous depression history did not impact the efficacy of the model for PPD recognition. Moreover, we found that the most predictive biomarker of PPD was calories burned during the basal metabolic rate. Finally, individualized models surpassed the performance of a conventional cohort-based model for PPD detection.

CONCLUSIONS

This research establishes consumer wearables as a promising tool for PPD identification and highlights personalized ML approaches, which could transform early disease detection strategies.

Machine Learning in the Parkinson's disease smartwatch (PADS) dataset

The utilisation of smart devices, such as smartwatches and smartphones, in the field of movement disorders research has gained significant attention. However, the absence of a comprehensive dataset with movement data and clinical annotations, encompassing a wide range of movement disorders including Parkinson's disease (PD) and its differential diagnoses (DD), presents a significant gap. The availability of such a dataset is crucial for the development of reliable machine learning (ML) models on smart devices, enabling the detection of diseases and monitoring of treatment efficacy in a home-based setting. We conducted a three-year cross-sectional study at a large tertiary care hospital. A multi-modal smartphone app integrated electronic questionnaires and smartwatch measures during an interactive assessment designed by neurologists to provoke subtle changes in movement pathologies. We captured over 5000 clinical assessment steps from 504 participants, including PD, DD, and healthy controls (HC). After age-matching, an integrative ML approach combining classical signal processing and advanced deep learning techniques was implemented and cross-validated. The models achieved an average balanced accuracy of 91.16% in the classification PD vs. HC, while PD vs. DD scored 72.42%. The numbers suggest promising performance while distinguishing similar disorders remains challenging. The extensive annotations, including details on demographics, medical history, symptoms, and movement steps, provide a comprehensive database to ML techniques and encourage further investigations into phenotypical biomarkers related to movement disorders.

An in-depth survey on Deep Learning-based Motor Imagery Electroencephalogram (EEG) classification

Electroencephalogram (EEG)-based Brain-Computer Interfaces (BCIs) build a communication path between human brain and external devices. Among EEG-based BCI paradigms, the most commonly used one is motor imagery (MI). As a hot research topic, MI EEG-based BCI has largely contributed to medical fields and smart home industry. However, because of the low signal-to-noise ratio (SNR) and the non-stationary characteristic of EEG data, it is difficult to correctly classify different types of MI-EEG signals. Recently, the advances in Deep Learning (DL) significantly facilitate the development of MI EEG-based BCIs. In this paper, we provide a systematic survey of DL-based MI-EEG classification methods. Specifically, we first comprehensively discuss several important aspects of DL-based MI-EEG classification, covering input formulations, network architectures, public datasets, etc. Then, we summarize problems in model performance comparison and give guidelines to future studies for fair performance comparison. Next, we fairly evaluate the representative DL-based models using source code released by the authors and meticulously analyse the evaluation results. By performing ablation study on the network architecture, we found that (1) effective feature fusion is indispensable for multi-stream CNN-based models. (2) LSTM should be combined with spatial feature extraction techniques to obtain good classification performance. (3) the use of dropout contributes little to improving the model performance, and that (4) adding fully connected layers to the models significantly increases their parameters but it might not improve their performance. Finally, we raise several open issues in MI-EEG classification and provide possible future research directions.

EEG-based classification of individuals with neuropsychiatric disorders using deep neural networks: A systematic review of current status and future directions

The use of deep neural networks for electroencephalogram (EEG) classification has rapidly progressed and gained popularity in recent years, but automatic feature extraction from EEG signals remains a challenging task. The classification of neuropsychiatric disorders demands the extraction of neuro-markers for use in automated EEG classification. Numerous advanced deep learning algorithms can be used for this purpose. In this article, we present a comprehensive review of the main factors and parameters that affect the performance of deep neural networks in classifying different neuropsychiatric disorders using EEG signals. We also analyze the EEG features used for improving classification performance. Our analysis includes 82 scientific journal papers that applied deep neural networks for subject-wise classification based on EEG signals. We extracted information on the EEG dataset and types of disorders, deep neural network structures, performance, and hyperparameters. The results show that most studies have focused on clinical classification, achieving an average accuracy of 91.83 ± 7.34, with convolutional neural networks (CNNs) being the most frequently used network architecture and resting-state EEG signals being the most commonly used data type. Additionally, the review reveals that depression (N = 18), Alzheimer's (N = 11), and schizophrenia (N = 11) were studied more frequently than other types of neuropsychiatric disorders. Our review provides insight into the performance of deep neural networks in EEG classification and highlights the importance of EEG feature extraction in improving classification accuracy. By identifying the main factors and parameters that affect deep neural network performance in EEG classification, our review can guide future research in this area. We hope that our findings will encourage further exploration of deep learning methods for EEG classification and contribute to the development of more accurate and effective methods for diagnosing and monitoring neuropsychiatric disorders using EEG signals.

Narrowing the gap: expected versus deployment performance

OBJECTIVES

Successful model development requires both an accurate a priori understanding of future performance and high performance on deployment. Optimistic estimations of model performance that are unrealized in real-world clinical settings can contribute to nonuse of predictive models. This study used 2 tasks, predicting ICU mortality and Bi-Level Positive Airway Pressure failure, to quantify: (1) how well internal test performances derived from different methods of partitioning data into development and test sets estimate future deployment performance of Recurrent Neural Network models and (2) the effects of including older data in the training set on models' performance.

MATERIALS AND METHODS

The cohort consisted of patients admitted between 2010 and 2020 to the Pediatric Intensive Care Unit of a large quaternary children's hospital. 2010-2018 data were partitioned into different development and test sets to measure internal test performance. Deployable models were trained on 2010-2018 data and assessed on 2019-2020 data, which was conceptualized to represent a real-world deployment scenario. Optimism, defined as the overestimation of the deployed performance by internal test performance, was measured. Performances of deployable models were also compared with each other to quantify the effect of including older data during training.

RESULTS, DISCUSSION, AND CONCLUSION

Longitudinal partitioning methods, where models are tested on newer data than the development set, yielded the least optimism. Including older years in the training dataset did not degrade deployable model performance. Using all available data for model development fully leveraged longitudinal partitioning by measuring year-to-year performance.

Milani A, Madani Tonekaboni SA, Rabbani H, Hekmatnia A, Kafieh R. Biased Deep Learning Methods in Detection of COVID-19 Using CT Images: A Challenge Mounted by Subject-Wise-Split ISFCT Dataset

Accurate detection of respiratory system damage including COVID-19 is considered one of the crucial applications of deep learning (DL) models using CT images. However, the main shortcoming of the published works has been unreliable reported accuracy and the lack of repeatability with new datasets, mainly due to slice-wise splits of the data, creating dependency between training and test sets due to shared data across the sets. We introduce a new dataset of CT images (ISFCT Dataset) with labels indicating the subject-wise split to train and test our DL algorithms in an unbiased manner. We also use this dataset to validate the real performance of the published works in a subject-wise data split. Another key feature provides more specific labels (eight characteristic lung features) rather than being limited to COVID-19 and healthy labels. We show that the reported high accuracy of the existing models on current slice-wise splits is not repeatable for subject-wise splits, and distribution differences between data splits are demonstrated using t-distribution stochastic neighbor embedding. We indicate that, by examining subject-wise data splitting, less complicated models show competitive results compared to the exiting complicated models, demonstrating that complex models do not necessarily generate accurate and repeatable results.

Performance of multiple neural networks in predicting lower limb joint moments using wearable sensors

Joint moment measurements represent an objective biomechemical parameter in joint health assessment. Inverse dynamics based on 3D motion capture data is the current 'gold standard' to estimate joint moments. Recently, machine learning combined with data measured by wearable technologies such electromyography (EMG), inertial measurement units (IMU), and electrogoniometers (GON) has been used to enable fast, easy, and low-cost measurements of joint moments. This study investigates the ability of various deep neural networks to predict lower limb joint moments merely from IMU sensors. The performance of five different deep neural networks (InceptionTimePlus, eXplainable convolutional neural network (XCM), XCMplus, Recurrent neural network (RNNplus), and Time Series Transformer (TSTPlus)) were tested to predict hip, knee, ankle, and subtalar moments using acceleration and gyroscope measurements of four IMU sensors at the trunk, thigh, shank, and foot. Multiple locomotion modes were considered including level-ground walking, treadmill walking, stair ascent, stair descent, ramp ascent, and ramp descent. We show that XCM can accurately predict lower limb joint moments using data of only four IMUs with RMSE of 0.046 ± 0.013 Nm/kg compared to 0.064 ± 0.003 Nm/kg on average for the other architectures. We found that hip, knee, and ankle joint moments predictions had a comparable RMSE with an average of 0.069 Nm/kg, while subtalar joint moments had the lowest RMSE of 0.033 Nm/kg. The real-time feedback that can be derived from the proposed method can be highly valuable for sports scientists and physiotherapists to gain insights into biomechanics, technique, and form to develop personalized training and rehabilitation programs.

A novel machine learning system for identifying sleep-wake states in mice

Research into sleep-wake behaviors relies on scoring sleep states, normally done by manual inspection of electroencephalogram (EEG) and electromyogram (EMG) recordings. This is a highly time-consuming process prone to inter-rater variability. When studying relationships between sleep and motor function, analyzing arousal states under a four-state system of active wake (AW), quiet wake (QW), nonrapid-eye-movement (NREM) sleep, and rapid-eye-movement (REM) sleep provides greater precision in behavioral analysis but is a more complex model for classification than the traditional three-state identification (wake, NREM, and REM sleep) usually used in rodent models. Characteristic features between sleep-wake states provide potential for the use of machine learning to automate classification. Here, we devised SleepEns, which uses a novel ensemble architecture, the time-series ensemble. SleepEns achieved 90% accuracy to the source expert, which was statistically similar to the performance of two other human experts. Considering the capacity for classification disagreements that are still physiologically reasonable, SleepEns had an acceptable performance of 99% accuracy, as determined blindly by the source expert. Classifications given by SleepEns also maintained similar sleep-wake characteristics compared to expert classifications, some of which were essential for sleep-wake identification. Hence, our approach achieves results comparable to human ability in a fraction of the time. This new machine-learning ensemble will significantly impact the ability of sleep researcher to detect and study sleep-wake behaviors in mice and potentially in humans.

Developing better digital health measures of Parkinson's disease using free living data and a crowdsourced data analysis challenge

One of the promising opportunities of digital health is its potential to lead to more holistic understandings of diseases by interacting with the daily life of patients and through the collection of large amounts of real-world data. Validating and benchmarking indicators of disease severity in the home setting is difficult, however, given the large number of confounders present in the real world and the challenges in collecting ground truth data in the home. Here we leverage two datasets collected from patients with Parkinson's disease, which couples continuous wrist-worn accelerometer data with frequent symptom reports in the home setting, to develop digital biomarkers of symptom severity. Using these data, we performed a public benchmarking challenge in which participants were asked to build measures of severity across 3 symptoms (on/off medication, dyskinesia, and tremor). 42 teams participated and performance was improved over baseline models for each subchallenge. Additional ensemble modeling across submissions further improved performance, and the top models validated in a subset of patients whose symptoms were observed and rated by trained clinicians.

Computerised prediction of healing for venous leg ulcers

Early prediction of delayed healing for venous leg ulcers could improve management outcomes by enabling earlier initiation of adjuvant therapies. In this paper, we propose a framework for computerised prediction of healing for venous leg ulcers assessed in home settings using thermal images of the 0 week. Wound data of 56 older participants over 12 weeks were used for the study. Thermal images of the wounds were collected in their homes and labelled as healed or unhealed at the 12th week follow up. Textural information of the thermal images at week 0 was extracted. Thermal images of unhealed wounds had a higher variation of grey tones distribution. We demonstrated that the first three principal components of the textural features from one timepoint can be used as an input to a Bayesian neural network to discriminate between healed and unhealed wounds. Using the optimal Bayesian neural network, the classification results showed 78.57% sensitivity and 60.00% specificity. This non-contact method, incorporating machine learning, can provide a computerised prediction of this delay in the first assessment (week 0) in participants' homes compared to the current method that is able to do this in 3rd week and requires contact digital planimetry.

Disentangling personalized treatment effects from “time-of-the-day” confounding in mobile health studies

Ideally, a patient’s response to medication can be monitored by measuring changes in performance of some activity. In observational studies, however, any detected association between treatment (“on-medication” vs “off-medication”) and the outcome (performance in the activity) might be due to confounders. In particular, causal inferences at the personalized level are especially vulnerable to confounding effects that arise in a cyclic fashion. For quick acting medications, effects can be confounded by circadian rhythms and daily routines. Using the time-of-the-day as a surrogate for these confounders and the performance measurements as captured on a smartphone, we propose a personalized statistical approach to disentangle putative treatment and “time-of-the-day” effects, that leverages conditional independence relations spanned by causal graphical models involving the treatment, time-of-the-day, and outcome variables. Our approach is based on conditional independence tests implemented via standard and temporal linear regression models. Using synthetic data, we investigate when and how residual autocorrelation can affect the standard tests, and how time series modeling (namely, ARIMA and robust regression via HAC covariance matrix estimators) can remedy these issues. In particular, our simulations illustrate that when patients perform their activities in a paired fashion, positive autocorrelation can lead to conservative results for the standard regression approach (i.e., lead to deflated true positive detection), whereas negative autocorrelation can lead to anticonservative behavior (i.e., lead to inflated false positive detection). The adoption of time series methods, on the other hand, leads to well controlled type I error rates. We illustrate the application of our methodology with data from a Parkinson’s disease mobile health study.

Detection of Patients with Congenital and Often Concealed Long-QT Syndrome by Novel Deep Learning Models

Introduction: The long-QT syndrome (LQTS) is the most common ion channelopathy, typically presenting with a prolonged QT interval and clinical symptoms such as syncope or sudden cardiac death. Patients may present with a concealed phenotype making the diagnosis challenging. Correctly diagnosing at-risk patients is pivotal to starting early preventive treatment. Objective: Identification of congenital and often concealed LQTS by utilizing novel deep learning network architectures, which are specifically designed for multichannel time series and therefore particularly suitable for ECG data. Design and Results: A retrospective artificial intelligence (AI)-based analysis was performed using a 12-lead ECG of genetically confirmed LQTS (n = 124), including 41 patients with a concealed LQTS (33%), and validated against a control cohort (n = 161 of patients) without known LQTS or without QT-prolonging drug treatment but any other cardiovascular disease. The performance of a fully convolutional network (FCN) used in prior studies was compared with a different, novel convolutional neural network model (XceptionTime). We found that the XceptionTime model was able to achieve a higher balanced accuracy score (91.8%) than the associated FCN metric (83.6%), indicating improved prediction possibilities of novel AI architectures. The predictive accuracy prevailed independently of age and QT_c parameters. Conclusions: In this study, the XceptionTime model outperformed the FCN model for LQTS patients with even better results than in prior studies. Even when a patient cohort with cardiovascular comorbidities is used. AI-based ECG analysis is a promising step for correct LQTS patient identification, especially if common diagnostic measures might be misleading.

S2 Heart Sound Detects Aortic Valve Calcification Independent of Hemodynamic Changes in Mice

Background

Calcific aortic valve disease (CAVD) is often undiagnosed in asymptomatic patients, especially in underserved populations. Although artificial intelligence has improved murmur detection in auscultation exams, murmur manifestation depends on hemodynamic factors that can be independent of aortic valve (AoV) calcium load and function. The aim of this study was to determine if the presence of AoV calcification directly influences the S2 heart sound.

Methods

Adult C57BL/6J mice were assigned to the following 12-week-long diets: (1) Control group (n = 11) fed a normal chow, (2) Adenine group (n = 4) fed an adenine-supplemented diet to induce chronic kidney disease (CKD), and (3) Adenine + HP (n = 9) group fed the CKD diet for 6 weeks, then supplemented with high phosphate (HP) for another 6 weeks to induce AoV calcification. Phonocardiograms, echocardiogram-based valvular function, and AoV calcification were assessed at endpoint.

Results

Mice on the Adenine + HP diet had detectable AoV calcification (9.28 ± 0.74% by volume). After segmentation and dimensionality reduction, S2 sounds were labeled based on the presence of disease: Healthy, CKD, or CKD + CAVD. The dataset (2,516 S2 sounds) was split subject-wise, and an ensemble learning-based algorithm was developed to classify S2 sound features. For external validation, the areas under the receiver operating characteristic curve of the algorithm to classify mice were 0.9940 for Healthy, 0.9717 for CKD, and 0.9593 for CKD + CAVD. The algorithm had a low misclassification performance of testing set S2 sounds (1.27% false positive, 1.99% false negative).

Conclusion

Our ensemble learning-based algorithm demonstrated the feasibility of using the S2 sound to detect the presence of AoV calcification. The S2 sound can be used as a marker to identify AoV calcification independent of hemodynamic changes observed in echocardiography.

Machine Learning-Based Diabetic Neuropathy and Previous Foot Ulceration Patients Detection Using Electromyography and Ground Reaction Forces during Gait

Diabetic neuropathy (DN) is one of the prevalent forms of neuropathy that involves alterations in biomechanical changes in the human gait. Diabetic foot ulceration (DFU) is one of the pervasive types of complications that arise due to DN. In the literature, for the last 50 years, researchers have been trying to observe the biomechanical changes due to DN and DFU by studying muscle electromyography (EMG) and ground reaction forces (GRF). However, the literature is contradictory. In such a scenario, we propose using Machine learning techniques to identify DN and DFU patients by using EMG and GRF data. We collected a dataset from the literature which involves three patient groups: Control (n = 6), DN (n = 6), and previous history of DFU (n = 9) and collected three lower limb muscles EMG (tibialis anterior (TA), vastus lateralis (VL), gastrocnemius lateralis (GL)), and three GRF components (GRFx, GRFy, and GRFz). Raw EMG and GRF signals were preprocessed, and different feature extraction techniques were applied to extract the best features from the signals. The extracted feature list was ranked using four different feature ranking techniques, and highly correlated features were removed. In this study, we considered different combinations of muscles and GRF components to find the best performing feature list for the identification of DN and DFU. We trained eight different conventional ML models: Discriminant analysis classifier (DAC), Ensemble classification model (ECM), Kernel classification model (KCM), k-nearest neighbor model (KNN), Linear classification model (LCM), Naive Bayes classifier (NBC), Support vector machine classifier (SVM), and Binary decision classification tree (BDC), to find the best-performing algorithm and optimized that model. We trained the optimized the ML algorithm for different combinations of muscles and GRF component features, and the performance matrix was evaluated. Our study found the KNN algorithm performed well in identifying DN and DFU, and we optimized it before training. We found the best accuracy of 96.18% for EMG analysis using the top 22 features from the chi-square feature ranking technique for features from GL and VL muscles combined. In the GRF analysis, the model showed 98.68% accuracy using the top 7 features from the Feature selection using neighborhood component analysis for the feature combinations from the GRFx-GRFz signal. In conclusion, our study has shown a potential solution for ML application in DN and DFU patient identification using EMG and GRF parameters. With careful signal preprocessing with strategic feature extraction from the biomechanical parameters, optimization of the ML model can provide a potential solution in the diagnosis and stratification of DN and DFU patients from the EMG and GRF signals.

Remote smartphone monitoring of Parkinson's disease and individual response to therapy

Remote health assessments that gather real-world data (RWD) outside clinic settings require a clear understanding of appropriate methods for data collection, quality assessment, analysis and interpretation. Here we examine the performance and limitations of smartphones in collecting RWD in the remote mPower observational study of Parkinson's disease (PD). Within the first 6 months of study commencement, 960 participants had enrolled and performed at least five self-administered active PD symptom assessments (speeded tapping, gait/balance, phonation or memory). Task performance, especially speeded tapping, was predictive of self-reported PD status (area under the receiver operating characteristic curve (AUC) = 0.8) and correlated with in-clinic evaluation of disease severity (r = 0.71; P < 1.8 × 10^-6) when compared with motor Movement Disorder Society-Unified Parkinson's Disease Rating Scale (MDS-UPDRS). Although remote assessment requires careful consideration for accurate interpretation of RWD, our results support the use of smartphones and wearables in objective and personalized disease assessments.

Sepsis Prediction for the General Ward Setting

Objective

To develop and evaluate a sepsis prediction model for the general ward setting and extend the evaluation through a novel pseudo-prospective trial design.

Design

Retrospective analysis of data extracted from electronic health records (EHR).

Setting

Single, tertiary-care academic medical center in St. Louis, MO, USA.

Patients

Adult, non-surgical inpatients admitted between January 1, 2012 and June 1, 2019.

Interventions

None.

Measurements and Main Results

Of the 70,034 included patient encounters, 3.1% were septic based on the Sepsis-3 criteria. Features were generated from the EHR data and were used to develop a machine learning model to predict sepsis 6-h ahead of onset. The best performing model had an Area Under the Receiver Operating Characteristic curve (AUROC or c-statistic) of 0.862 ± 0.011 and Area Under the Precision-Recall Curve (AUPRC) of 0.294 ± 0.021 compared to that of Logistic Regression (0.857 ± 0.008 and 0.256 ± 0.024) and NEWS 2 (0.699 ± 0.012 and 0.092 ± 0.009). In the pseudo-prospective trial, 388 (69.7%) septic patients were alerted on with a specificity of 81.4%. Within 24 h of crossing the alert threshold, 20.9% had a sepsis-related event occur.

Conclusions

A machine learning model capable of predicting sepsis in the general ward setting was developed using the EHR data. The pseudo-prospective trial provided a more realistic estimation of implemented performance and demonstrated a 29.1% Positive Predictive Value (PPV) for sepsis-related intervention or outcome within 48 h.

Heterogeneous digital biomarker integration out-performs patient self-reports in predicting Parkinson's disease

Parkinson's disease (PD) is one of the first diseases where digital biomarkers demonstrated excellent performance in differentiating disease from healthy individuals. However, no study has systematically compared and leveraged multiple types of digital biomarkers to predict PD. Particularly, machine learning works on the fine-motor skills of PD are limited. Here, we developed deep learning methods that achieved an AUC (Area Under the receiver operator characteristic Curve) of 0.933 in identifying PD patients on 6418 individuals using 75048 tapping accelerometer and position records. Performance of tapping is superior to gait/rest and voice-based models obtained from the same benchmark population. Assembling the three models achieved a higher AUC of 0.944. Notably, the models not only correlated strongly to, but also performed better than patient self-reported symptom scores in diagnosing PD. This study demonstrates the complementary predictive power of tapping, gait/rest and voice data and establishes integrative deep learning-based models for identifying PD.

Predicting continuous ground reaction forces from accelerometers during uphill and downhill running: a recurrent neural network solution

BACKGROUND

Ground reaction forces (GRFs) are important for understanding human movement, but their measurement is generally limited to a laboratory environment. Previous studies have used neural networks to predict GRF waveforms during running from wearable device data, but these predictions are limited to the stance phase of level-ground running. A method of predicting the normal (perpendicular to running surface) GRF waveform using wearable devices across a range of running speeds and slopes could allow researchers and clinicians to predict kinetic and kinematic variables outside the laboratory environment.

PURPOSE

We sought to develop a recurrent neural network capable of predicting continuous normal (perpendicular to surface) GRFs across a range of running speeds and slopes from accelerometer data.

METHODS

Nineteen subjects ran on a force-measuring treadmill at five slopes (0°, ±5°, ±10°) and three speeds (2.5, 3.33, 4.17 m/s) per slope with sacral- and shoe-mounted accelerometers. We then trained a recurrent neural network to predict normal GRF waveforms frame-by-frame. The predicted versus measured GRF waveforms had an average ± SD RMSE of 0.16 ± 0.04 BW and relative RMSE of 6.4 ± 1.5% across all conditions and subjects.

RESULTS

The recurrent neural network predicted continuous normal GRF waveforms across a range of running speeds and slopes with greater accuracy than neural networks implemented in previous studies. This approach may facilitate predictions of biomechanical variables outside the laboratory in near real-time and improves the accuracy of quantifying and monitoring external forces experienced by the body when running.

A machine learning approach for semi-automatic assessment of IADL dependence in older adults with wearable sensors

BACKGROUND AND OBJECTIVE

The assessment of dependence in older adults currently requires a manual collection of data taken from questionnaires. This process is time consuming for the clinicians and intrudes the daily life of the elderly. This paper aims to semi-automate the acquisition and analysis of health data to assess and predict the dependence in older adults while executing one instrumental activity of daily living (IADL).

METHODS

In a mobile-health (m-health) scenario, we analyze whether the acquisition of data through wearables during the performance of IADLs, and with the help of machine learning techniques could replace the traditional questionnaires to evaluate dependence. To that end, we collected data from wearables, while older adults do the shopping activity. A trial supervisor (TS) labelled the different shopping stages (SS) in the collected data. We performed data pre-processing techniques over those SS and analyzed them with three machine learning algorithms: k-Nearest Neighbors (k-NN), Random Forest (RF) and Support Vector Machines (SVM).

RESULTS

Our results confirm that it is possible to replace the traditional questionnaires with wearable data. In particular, the best learning algorithm we tried reported an accuracy of 97% in the assessment of dependence. We tuned the hyperparameters of this algorithm and used embedded feature selection technique to get the best performance with a subset of only 10 features out of the initial 85. This model considers only features extracted from four sensors of a single wearable: accelerometer, heart rate, electrodermal activity and temperature. Although these features are not observational, our current proposal is semi-automatic, because it needs a TS labelling the SS (with a smartphone application). In the future, this labelling process could be automatic as well.

CONCLUSIONS

Our method can semi-automatically assess the dependence, without disturbing daily activities of elderly people. This method can save clinicians' time in the evaluation of dependence in older adults and reduce healthcare costs.

Comorbid insomnia among breast cancer survivors and its prediction using machine learning: a nationwide study in Japan

OBJECTIVE

Insomnia is an increasingly recognized major symptom of breast cancer which can seriously disrupt the quality of life during and many years after treatment. Sleep problems have also been linked with survival in women with breast cancer. The aims of this study were to estimate the prevalence of insomnia in breast cancers survivors, clarify the clinical characteristics of their sleep difficulties and use machine learning techniques to explore clinical insights.

METHODS

Our analysis of data, obtained in a nationwide questionnaire survey of breast cancer survivors in Japan, revealed a prevalence of suspected insomnia of 37.5%. With the clinical data obtained, we then used machine learning algorithms to develop a classifier that predicts comorbid insomnia. The performance of the prediction model was evaluated using 8-fold cross-validation.

RESULTS

When using optimal hyperparameters, the L2 penalized logistic regression model and the XGBoost model provided predictive accuracy of 71.5 and 70.6% for the presence of suspected insomnia, with areas under the curve of 0.76 and 0.75, respectively. Population segments with high risk of insomnia were also extracted using the RuleFit algorithm. We found that cancer-related fatigue is a predictor of insomnia in breast cancer survivors.

CONCLUSIONS

The high prevalence of sleep problems and its link with mortality warrants routine screening. Our novel predictive model using a machine learning approach offers clinically important insights for the early detection of comorbid insomnia and intervention in breast cancer survivors.

Subject-Specific Cognitive Workload Classification Using EEG-Based Functional Connectivity and Deep Learning

Cognitive workload is a crucial factor in tasks involving dynamic decision-making and other real-time and high-risk situations. Neuroimaging techniques have long been used for estimating cognitive workload. Given the portability, cost-effectiveness and high time-resolution of EEG as compared to fMRI and other neuroimaging modalities, an efficient method of estimating an individual’s workload using EEG is of paramount importance. Multiple cognitive, psychiatric and behavioral phenotypes have already been known to be linked with “functional connectivity”, i.e., correlations between different brain regions. In this work, we explored the possibility of using different model-free functional connectivity metrics along with deep learning in order to efficiently classify the cognitive workload of the participants. To this end, 64-channel EEG data of 19 participants were collected while they were doing the traditional n-back task. These data (after pre-processing) were used to extract the functional connectivity features, namely Phase Transfer Entropy (PTE), Mutual Information (MI) and Phase Locking Value (PLV). These three were chosen to do a comprehensive comparison of directed and non-directed model-free functional connectivity metrics (allows faster computations). Using these features, three deep learning classifiers, namely CNN, LSTM and Conv-LSTM were used for classifying the cognitive workload as low (1-back), medium (2-back) or high (3-back). With the high inter-subject variability in EEG and cognitive workload and recent research highlighting that EEG-based functional connectivity metrics are subject-specific, subject-specific classifiers were used. Results show the state-of-the-art multi-class classification accuracy with the combination of MI with CNN at 80.87%, followed by the combination of PLV with CNN (at 75.88%) and MI with LSTM (at 71.87%). The highest subject specific performance was achieved by the combinations of PLV with Conv-LSTM, and PLV with CNN with an accuracy of 97.92%, followed by the combination of MI with CNN (at 95.83%) and MI with Conv-LSTM (at 93.75%). The results highlight the efficacy of the combination of EEG-based model-free functional connectivity metrics and deep learning in order to classify cognitive workload. The work can further be extended to explore the possibility of classifying cognitive workload in real-time, dynamic and complex real-world scenarios.

Patient Onboarding and Engagement to Build a Digital Registry after Enrollment in a Clinical Trial: Results of the TAILOR-PCI Digital Study (Preprint)

Background

The Tailored Antiplatelet Initiation to Lessen Outcomes Due to Decreased Clopidogrel Response After Percutaneous Coronary Intervention (TAILOR-PCI) Digital Study is a novel proof-of-concept study that evaluated the feasibility of extending the TAILOR-PCI randomized controlled trial (RCT) follow-up period by using a remote digital platform.

Objective

The aim of this study is to describe patients’ onboarding, engagement, and results in a digital study after enrollment in an RCT.

Methods

In this intervention study, previously enrolled TAILOR-PCI patients in the United States and Canada within 24 months of randomization were invited by letter to download the study app. Those who did not respond to the letter were contacted by phone to survey the reasons for nonparticipation. A direct-to-patient digital research platform (the Eureka Research Platform) was used to onboard patients, obtain consent, and administer activities in the digital study. The patients were asked to complete health-related surveys and digitally provide follow-up data. Our primary end points were the consent rate, the duration of participation, and the monthly activity completion rate in the digital study. The hypothesis being tested was formulated before data collection began.

Results

After the parent trial was completed, letters were mailed to 907 eligible patients (representing 18.8% [907/4837] of total enrolled in the RCT) within 15.6 (SD 5.2) months of randomization across 24 sites. Among the 907 patients invited, 290 (32%) visited the study website and 110 (12.1%) consented—40.9% (45/110) after the letter, 33.6% (37/110) after the first phone call, and 25.5% (28/110) after the second call. Among the 47.4% (409/862) of patients who responded, 41.8% (171/409) declined to participate because of a lack of time, 31.2% (128/409) declined because of the lack of a smartphone, and 11.5% (47/409) declined because of difficulty understanding what was expected of them in the study. Patients who consented were older (aged 65.3 vs 62.5 years; P=.006) and had a lower prevalence of diabetes (19% vs 30%; P=.02) or tobacco use (6.4% vs 24.8%; P<.001). A greater proportion had bachelor’s degrees (47.2% vs 25.7%; P<.001) and were more computer literate (90.5% vs 62.3% of daily internet use; P<.001) than those who did not consent. The average completion rate of the 920 available monthly electronic visits was 64.9% (SD 7.6%); there was no decrease in this rate throughout the study duration.

Conclusions

Extended follow-up after enrollment in an RCT by using a digital study was technically feasible but was limited because of the inability to contact most eligible patients or a lack of time or access to a smartphone. Among the enrolled patients, most completed the required electronic visits. Enhanced recruitment methods, such as the introduction of a digital study at the time of RCT consent, smartphone provision, and robust study support for onboarding, should be explored further.

Trial Registration

Clinicaltrails.gov NCT01742117; https://clinicaltrials.gov/ct2/show/NCT01742117

Whole blood gene expression within days after total-body irradiation predicts long term survival in Gottingen minipigs

Gottingen minipigs mirror the physiological radiation response observed in humans and hence make an ideal candidate model for studying radiation biodosimetry for both limited-sized and mass casualty incidents. We examined the whole blood gene expression profiles starting one day after total-body irradiation with increasing doses of gamma-rays. The minipigs were monitored for up to 45 days or time to euthanasia necessitated by radiation effects. We successfully identified dose- and time-agnostic (over a 1-7 day period after radiation), survival-predictive gene expression signatures derived using machine-learning algorithms with high sensitivity and specificity. These survival-predictive signatures fare better than an optimally performing dose-differentiating signature or blood cellular profiles. These findings suggest that prediction of survival is a much more useful parameter for making triage, resource-utilization and treatment decisions in a resource-constrained environment compared to predictions of total dose received. It should hopefully be possible to build such classifiers for humans in the future.

Detecting Mild Cognitive Impairment via Digital Biomarkers of Cognitive Performance Found in Klondike Solitaire: A Machine-Learning Study

Background

Mild cognitive impairment (MCI) is a condition that entails a slight yet noticeable decline in cognition that exceeds normal age-related changes. Older adults living with MCI have a higher chance of progressing to dementia, which warrants regular cognitive follow-up at memory clinics. However, due to time and resource constraints, this follow-up is conducted at separate moments in time with large intervals in between. Casual games, embedded into the daily life of older adults, may prove to be a less resource-intensive medium that yields continuous and rich data on a patient's cognition.

Objective

To explore whether digital biomarkers of cognitive performance, found in the casual card game Klondike Solitaire, can be used to train machine-learning models to discern games played by older adults living with MCI from their healthy counterparts.

Methods

Digital biomarkers of cognitive performance were captured from 23 healthy older adults and 23 older adults living with MCI, each playing 3 games of Solitaire with 3 different deck shuffles. These 3 deck shuffles were identical for each participant. Using a supervised stratified, 5-fold, cross-validated, machine-learning procedure, 19 different models were trained and optimized for F1 score.

Results

The 3 best performing models, an Extra Trees model, a Gradient Boosting model, and a Nu-Support Vector Model, had a cross-validated F1 training score on the validation set of ≥0.792. The F1 score and AUC of the test set were, respectively, >0.811 and >0.877 for each of these models. These results indicate psychometric properties comparative to common cognitive screening tests.

Conclusion

The results suggest that commercial card games, not developed to address specific mental processes, may be used for measuring cognition. The digital biomarkers derived from Klondike Solitaire show promise and may prove useful to fill the current blind spot between consultations.

Radiomic and Genomic Machine Learning Method Performance for Prostate Cancer Diagnosis: Systematic Literature Review

BACKGROUND

Machine learning algorithms have been drawing attention at the joining of pathology and radiology in prostate cancer research. However, due to their algorithmic learning complexity and the variability of their architecture, there is an ongoing need to analyze their performance.

OBJECTIVE

This study assesses the source of heterogeneity and the performance of machine learning applied to radiomic, genomic, and clinical biomarkers for the diagnosis of prostate cancer. One research focus of this study was on clearly identifying problems and issues related to the implementation of machine learning in clinical studies.

METHODS

Following the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) protocol, 816 titles were identified from the PubMed, Scopus, and OvidSP databases. Studies that used machine learning to detect prostate cancer and provided performance measures were included in our analysis. The quality of the eligible studies was assessed using the QUADAS-2 (quality assessment of diagnostic accuracy studies-version 2) tool. The hierarchical multivariate model was applied to the pooled data in a meta-analysis. To investigate the heterogeneity among studies, I2 statistics were performed along with visual evaluation of coupled forest plots. Due to the internal heterogeneity among machine learning algorithms, subgroup analysis was carried out to investigate the diagnostic capability of machine learning systems in clinical practice.

RESULTS

In the final analysis, 37 studies were included, of which 29 entered the meta-analysis pooling. The analysis of machine learning methods to detect prostate cancer reveals the limited usage of the methods and the lack of standards that hinder the implementation of machine learning in clinical applications.

CONCLUSIONS

The performance of machine learning for diagnosis of prostate cancer was considered satisfactory for several studies investigating the multiparametric magnetic resonance imaging and urine biomarkers; however, given the limitations indicated in our study, further studies are warranted to extend the potential use of machine learning to clinical settings. Recommendations on the use of machine learning techniques were also provided to help researchers to design robust studies to facilitate evidence generation from the use of radiomic and genomic biomarkers.

Crowdsourcing digital health measures to predict Parkinson's disease severity: the Parkinson's Disease Digital Biomarker DREAM Challenge

Consumer wearables and sensors are a rich source of data about patients' daily disease and symptom burden, particularly in the case of movement disorders like Parkinson's disease (PD). However, interpreting these complex data into so-called digital biomarkers requires complicated analytical approaches, and validating these biomarkers requires sufficient data and unbiased evaluation methods. Here we describe the use of crowdsourcing to specifically evaluate and benchmark features derived from accelerometer and gyroscope data in two different datasets to predict the presence of PD and severity of three PD symptoms: tremor, dyskinesia, and bradykinesia. Forty teams from around the world submitted features, and achieved drastically improved predictive performance for PD status (best AUROC = 0.87), as well as tremor- (best AUPR = 0.75), dyskinesia- (best AUPR = 0.48) and bradykinesia-severity (best AUPR = 0.95).

Smartphone- and Smartwatch-Based Remote Characterisation of Ambulation in Multiple Sclerosis During the Two-Minute Walk Test

Leveraging consumer technology such as smartphone and smartwatch devices to objectively assess people with multiple sclerosis (PwMS) remotely could capture unique aspects of disease progression. This study explores the feasibility of assessing PwMS and Healthy Control's (HC) physical function by characterising gait-related features, which can be modelled using machine learning (ML) techniques to correctly distinguish subgroups of PwMS from healthy controls. A total of 97 subjects (24 HC subjects, 52 mildly disabled (PwMSmild, EDSS [0-3]) and 21 moderately disabled (PwMSmod, EDSS [3.5-5.5]) contributed data which was recorded from a Two-Minute Walk Test (2MWT) performed out-of-clinic and daily over a 24-week period. Signal-based features relating to movement were extracted from sensors in smartphone and smartwatch devices. A large number of features (n = 156) showed fair-to-strong (R 0.3) correlations with clinical outcomes. LASSO feature selection was applied to select and rank subsets of features used for dichotomous classification between subject groups, which were compared using Logistic Regression (LR), Support Vector Machines (SVM) and Random Forest (RF) models. Classifications of subject types were compared using data obtained from smartphone, smartwatch and the fusion of features from both devices. Models built on smartphone features alone achieved the highest classification performance, indicating that accurate and remote measurement of the ambulatory characteristics of HC and PwMS can be achieved with only one device. It was observed however that smartphone-based performance was affected by inconsistent placement location (running belt versus pocket). Results show that PwMSmod could be distinguished from HC subjects (Acc. 82.2 ± 2.9%, Sen. 80.1 ± 3.9%, Spec. 87.2 ± 4.2%, F ₁ 84.3 ± 3.8), and PwMSmild (Acc. 82.3 ± 1.9%, Sen. 71.6 ± 4.2%, Spec. 87.0 ± 3.2%, F 1 75.1 ± 2.2) using an SVM classifier with a Radial Basis Function (RBF). PwMSmild were shown to exhibit HC-like behaviour and were thus less distinguishable from HC (Acc. 66.4 ± 4.5%, Sen. 67.5 ± 5.7%, Spec. 60.3 ± 6.7%, F ₁ 58.6 ± 5.8). Finally, it was observed that subjects in this study demonstrated low intra- and high inter-subject variability which was representative of subject-specific gait characteristics.

Identifying the signature of prospective motor control in children with autism

Failure to develop prospective motor control has been proposed to be a core phenotypic marker of autism spectrum disorders (ASD). However, whether genuine differences in prospective motor control permit discriminating between ASD and non-ASD profiles over and above individual differences in motor output remains unclear. Here, we combined high precision measures of hand movement kinematics and rigorous machine learning analyses to determine the true power of prospective movement data to differentiate children with autism and typically developing children. Our results show that while movement is unique to each individual, variations in the kinematic patterning of sequential grasping movements genuinely differentiate children with autism from typically developing children. These findings provide quantitative evidence for a prospective motor control impairment in autism and indicate the potential to draw inferences about autism on the basis of movement kinematics.

Design of a virtual longitudinal observational study in Parkinson's disease (AT-HOME PD)

OBJECTIVE

The expanding power and accessibility of personal technology provide an opportunity to reduce burdens and costs of traditional clinical site-centric therapeutic trials in Parkinson's disease and generate novel insights. The value of this approach has never been more evident than during the current COVID-19 pandemic. We sought to (1) establish and implement the infrastructure for longitudinal, virtual follow-up of clinical trial participants, (2) compare changes in smartphone-based assessments, online patient-reported outcomes, and remote expert assessments, and (3) explore novel digital markers of Parkinson's disease disability and progression.

METHODS

Participants from two recently completed phase III clinical trials of inosine and isradipine enrolled in Assessing Tele-Health Outcomes in Multiyear Extensions of Parkinson's Disease trials (AT-HOME PD), a two-year virtual cohort study. After providing electronic informed consent, individuals complete annual video visits with a movement disorder specialist, smartphone-based assessments of motor function and socialization, and patient-reported outcomes online.

RESULTS

From the two clinical trials, 226 individuals from 42 states in the United States and Canada enrolled. Of these, 181 (80%) have successfully downloaded the study's smartphone application and 161 (71%) have completed patient-reported outcomes on the online platform.

INTERPRETATION

It is feasible to conduct a large-scale, international virtual observational study following the completion of participation in brick-and-mortar clinical trials in Parkinson's disease. This study, which brings research to participants, will compare established clinical endpoints with novel digital biomarkers and thereby inform the longitudinal follow-up of clinical trial participants and design of future clinical trials.

Foot Strike Angle Prediction and Pattern Classification Using LoadsolTM Wearable Sensors: A Comparison of Machine Learning Techniques

The foot strike pattern performed during running is an important variable for runners, performance practitioners, and industry specialists. Versatile, wearable sensors may provide foot strike information while encouraging the collection of diverse information during ecological running. The purpose of the current study was to predict foot strike angle and classify foot strike pattern from Loadsol^TM wearable pressure insoles using three machine learning techniques (multiple linear regression―MR, conditional inference tree―TREE, and random forest―FRST). Model performance was assessed using three-dimensional kinematics as a ground-truth measure. The prediction-model accuracy was similar for the regression, inference tree, and random forest models (RMSE: MR = 5.16°, TREE = 4.85°, FRST = 3.65°; MAPE: MR = 0.32°, TREE = 0.45°, FRST = 0.33°), though the regression and random forest models boasted lower maximum precision (13.75° and 14.3°, respectively) than the inference tree (19.02°). The classification performance was above 90% for all models (MR = 90.4%, TREE = 93.9%, and FRST = 94.1%). There was an increased tendency to misclassify mid foot strike patterns in all models, which may be improved with the inclusion of more mid foot steps during model training. Ultimately, wearable pressure insoles in combination with simple machine learning techniques can be used to predict and classify a runner’s foot strike with sufficient accuracy.

Classification of Alpine Skiing Styles Using GNSS and Inertial Measurement Units

In alpine skiing, four commonly used turning styles are snowplow, snowplow-steering, drifting and carving. They differ significantly in speed, directional control and difficulty to execute. While they are visually distinguishable, data-driven classification is underexplored. The aim of this work is to classify alpine skiing styles based on a global navigation satellite system (GNSS) and inertial measurement units (IMU). Data of 2000 turns of 20 advanced or expert skiers were collected with two IMU sensors on the upper cuff of each ski boot and a mobile phone with GNSS. After feature extraction and feature selection, turn style classification was applied separately for parallel (drifted or carved) and non-parallel (snowplow or snowplow-steering) turns. The most important features for style classification were identified via recursive feature elimination. Three different classification methods were then tested and compared: Decision trees, random forests and gradient boosted decision trees. Classification accuracies were lowest for the decision tree and similar for the random forests and gradient boosted classification trees, which both achieved accuracies of more than 93% in the parallel classification task and 88% in the non-parallel case. While the accuracy might be improved by considering slope and weather conditions, these first results suggest that IMU data can classify alpine skiing styles reasonably well.