Evaluation of machine learning algorithms for health and wellness applications: A tutorial

Explainable deep-learning-based ischemia detection using hybrid O-15 H2O perfusion positron emission tomography and computed tomography imaging with clinical data

BACKGROUND

We developed an explainable deep-learning (DL)-based classifier to identify flow-limiting coronary artery disease (CAD) by O-15 H2O perfusion positron emission tomography computed tomography (PET/CT) and coronary CT angiography (CTA) imaging. The classifier uses polar map images with numerical data and visualizes data findings.

METHODS

A DLmodel was implemented and evaluated on 138 individuals, consisting of a combined image-and data-based classifier considering 35 clinical, CTA, and PET variables. Data from invasive coronary angiography were used as reference. Performance was evaluated with clinical classification using accuracy (ACC), area under the receiver operating characteristic curve (AUC), F1 score (F1S), sensitivity (SEN), specificity (SPE), precision (PRE), net benefit, and Cohen's Kappa. Statistical testing was conducted using McNemar's test.

RESULTS

The DL model had a median ACC = 0.8478, AUC = 0.8481, F1S = 0.8293, SEN = 0.8500, SPE = 0.8846, and PRE = 0.8500. Improved detection of true-positive and false-negative cases, increased net benefit in thresholds up to 34%, and comparable Cohen's kappa was seen, reaching similar performance to clinical reading. Statistical testing revealed no significant differences between DL model and clinical reading.

CONCLUSIONS

The combined DL model is a feasible and an effective method in detection of CAD, allowing to highlight important data findings individually in interpretable manner.

Two-Stage Classification of Future Knee Osteoarthritis Severity After 8 Years Using MRI: Data from the Osteoarthritis Initiative

Currently, there are no methods or tools available in clinical practice for classifying future knee osteoarthritis (KOA). In this study, we aimed to fill this gap by classifying future KOA into three severity grades: KL01 (healthy), KL2 (moderate), and KL34 (severe) based on the Kellgren-Lawrance scale. Due to the complex nature of multiclass classification, we used a two-stage method, which separates the classification task into two binary classifications (KL01 vs. KL234 in the first stage and KL2 vs. KL34 in the second stage). Our machine learning (ML) model used two Balanced Random Forest algorithms and was trained with gender, age, height, weight, and quantitative knee morphology obtained from magnetic resonance imaging. Our training dataset comprised longitudinal 8-year follow-up data of 1213 knees from the Osteoarthritis Initiative. Through extensive experimentation with various feature combinations, we identified KL baseline and weight as the most essential features, while gender surprisingly proved to be one of the least influential feature. Our best classification model generated a weighted F1 score of 79.0% and a balanced accuracy of 65.9%. The area under the receiver operating characteristic curve was 83.0% for healthy (KL01) versus moderate (KL2) or severe (KL34) KOA patients and 86.6% for moderate (KL2) versus severe (KL34) KOA patients. We found a statistically significant difference in performance between our two-stage classification model and the traditional single-stage classification model. These findings demonstrate the encouraging results of our two-stage classification model for multiclass KOA severity classification, suggesting its potential application in clinical settings in future.

Sex estimation from maxillofacial radiographs using a deep learning approach

The purpose of this study was to construct deep learning models for more efficient and reliable sex estimation. Two deep learning models, VGG16 and DenseNet-121, were used in this retrospective study. In total, 600 lateral cephalograms were analyzed. A saliency map was generated by gradient-weighted class activation mapping for each output. The two deep learning models achieved high values in each performance metric according to accuracy, sensitivity (recall), precision, F1 score, and areas under the receiver operating characteristic curve. Both models showed substantial differences in the positions indicated in saliency maps for male and female images. The positions in saliency maps also differed between VGG16 and DenseNet-121, regardless of sex. This analysis of our proposed system suggested that sex estimation from lateral cephalograms can be achieved with high accuracy using deep learning.

Exploring the mystery of colon cancer from the perspective of molecular subtypes and treatment

The molecular categorization of colon cancer patients remains elusive. Gene set enrichment analysis (GSEA), which investigates the dysregulated genes among tumor and normal samples, has revealed the pivotal role of epithelial-to-mesenchymal transition (EMT) in colon cancer pathogenesis. In this study, we employed multi-clustering method for grouping data, resulting in the identification of two clusters characterized by varying prognostic outcomes. These two subgroups not only displayed disparities in overall survival (OS) but also manifested variations in clinical variables, genetic mutation, and gene expression profiles. Using the nearest template prediction (NTP) method, we were able to replicate the molecular classification effectively within the original dataset and validated it across multiple independent datasets, underscoring its robust repeatability. Furthermore, we constructed two prognostic signatures tailored to each of these subgroups. Our molecular classification, centered on EMT, hold promise in offering fresh insights into the therapy strategies and prognosis assessment for colon cancer.

Evaluation metrics and statistical tests for machine learning

Research on different machine learning (ML) has become incredibly popular during the past few decades. However, for some researchers not familiar with statistics, it might be difficult to understand how to evaluate the performance of ML models and compare them with each other. Here, we introduce the most common evaluation metrics used for the typical supervised ML tasks including binary, multi-class, and multi-label classification, regression, image segmentation, object detection, and information retrieval. We explain how to choose a suitable statistical test for comparing models, how to obtain enough values of the metric for testing, and how to perform the test and interpret its results. We also present a few practical examples about comparing convolutional neural networks used to classify X-rays with different lung infections and detect cancer tumors in positron emission tomography images.

Applicability of machine learning algorithm to predict the therapeutic intervention success in Brazilian smokers

Smoking cessation is an important public health policy worldwide. However, as far as we know, there is a lack of screening of variables related to the success of therapeutic intervention (STI) in Brazilian smokers by machine learning (ML) algorithms. To address this gap in the literature, we evaluated the ability of eight ML algorithms to correctly predict the STI in Brazilian smokers who were treated at a smoking cessation program in Brazil between 2006 and 2017. The dataset was composed of 12 variables and the efficacies of the algorithms were measured by accuracy, sensitivity, specificity, positive predictive value (PPV) and area under the receiver operating characteristic curve. We plotted a decision tree flowchart and also measured the odds ratio (OR) between each independent variable and the outcome, and the importance of the variable for the best model based on PPV. The mean global values for the metrics described above were, respectively, 0.675±0.028, 0.803±0.078, 0.485±0.146, 0.705±0.035 and 0.680±0.033. Supporting vector machines performed the best algorithm with a PPV of 0.726±0.031. Smoking cessation drug use was the roof of decision tree with OR of 4.42 and importance of variable of 100.00. Increase in the number of relapses also promoted a positive outcome, while higher consumption of cigarettes resulted in the opposite. In summary, the best model predicted 72.6% of positive outcomes correctly. Smoking cessation drug use and higher number of relapses contributed to quit smoking, while higher consumption of cigarettes showed the opposite effect. There are important strategies to reduce the number of smokers and increase STI by increasing services and drug treatment for smokers.

On the bias in the AUC variance estimate

The area under the Receiver Operating Characteristic (ROC) curve (AUC) is a standard metric for quantifying and comparing binary classifiers. A popular approach to estimating the AUCs and the associated variabilities - the variance of the AUC or the full covariance matrix of multiple correlated AUCs - is the one proposed by DeLong et al [1], which is based on the Mann Whitney two-sample U-statistics. The bias of a variance estimator is an important factor in applications such as hypothesis testing and construction of confidence intervals - a negatively biased variance estimator may lead to incorrect conclusions, and a positive bias is conservative hence preferable. In this work, we show that the (co-)variance estimate in DeLong's approach is always positively biased. More specifically, the difference matrix between the expectation of the estimated covariance and the true covariance is a positive semi-definite matrix. This bias is non-negligible when the sample size is small, and quickly diminishes as the sample size increases. Our method relies on constructing, from the AUC kernel, a random variable whose (co-)variance matrix coincides with the bias, thereby establishing the claim. We also discuss alternative approaches to AUC variance estimation that may potentially reduce the bias.

Time-domain heart rate variability features for automatic congestive heart failure prediction

AIMS

Heart failure is a serious condition that often goes undiagnosed in primary care due to the lack of reliable diagnostic tools and the similarity of its symptoms with other diseases. Non-invasive monitoring of heart rate variability (HRV), which reflects the activity of the autonomic nervous system, could offer a novel and accurate way to detect and manage heart failure patients. This study aimed to assess the feasibility of using machine learning techniques on HRV data as a non-invasive biomarker to classify healthy adults and those with heart failure.

METHODS AND RESULTS

We used digitized electrocardiogram recordings from 54 adults with normal sinus rhythm and 44 adults categorized into New York Heart Association classes 1, 2, and 3, suffering from congestive heart failure. All recordings were sourced from the PhysioNet database. Following data pre-processing, we performed time-domain HRV analysis on all individual recordings, including root mean square of the successive difference in adjacent RR interval (RRi) (RMSSD), the standard deviation of RRi (SDNN, the NN stands for natural or sinus intervals), the standard deviation of the successive differences between successive RRi (SDSD), the number or percentage of RRi longer than 50 ms (NN50 and pNN50), and the average value of RRi [mean RR interval (mRRi)]. In our experimental classification performance evaluation, on the computed HRV parameters, we optimized hyperparameters and performed five-fold cross-validation using four machine learning classification algorithms: support vector machine, k-nearest neighbour (KNN), naïve Bayes, and decision tree (DT). We evaluated the prediction accuracy of these models using performance criteria, namely, precision, recall, specificity, F1 score, and overall accuracy. For added insight, we also presented receiver operating characteristic (ROC) plots and area under the ROC curve (AUC) values. The overall best performance accuracy of 77% was achieved when KNN and DT were trained on computed HRV parameters with a 5 min time window. KNN obtained an AUC of 0.77, while DT attained 0.78. Additionally, in the classification of severe congestive heart failure, KNN and DT had the best accuracy of 91%, with KNN achieving an AUC of 0.88 and DT obtaining 0.92.

CONCLUSIONS

The results show that HRV can accurately predict severe congestive heart failure. The findings of this study could inform the use of machine learning approaches on non-invasive HRV, to screen congestive heart failure individuals in primary care.

Investigation of different ML approaches in classification of emotions induced by acute stress

Background

Machine learning is becoming a common tool in monitoring emotion. However, methodological studies of the processing pipeline are scarce, especially ones using subjective appraisals as ground truth.

New method

A novel protocol was used to induce cognitive load and physical discomfort, and emotional dimensions (arousal, valence, and dominance) were reported after each task. The performance of five common ML models with a versatile set of features (physiological features, task performance data, and personality trait) was compared in binary classification of subjectively assessed emotions.

Results

The psychophysiological responses proved the protocol was successful in changing the mental state from baseline, also the cognitive and physical tasks were different. The optimization and performance of ML models used for emotion detection were evaluated. Additionally, methods to account for imbalanced classes were applied and shown to improve the classification performance.

Comparison with existing methods

Classification of human emotional states often assumes the states are determined by the stimuli. However, individual appraisals vary. None of the past studies have classified subjective emotional dimensions with a set of features including biosignals, personality and behavior.

Conclusion

Our data represent a typical setup in affective computing utilizing psychophysiological monitoring: N is low compared to number of features, inter-individual variability is high, and class imbalance cannot be avoided. Our observations are a) if possible, include features representing physiology, behavior and personality, b) use simple models and limited number of features to improve interpretability, c) address the possible imbalance, d) if the data size allows, use nested cross-validation.

Practical Considerations and Applied Examples of Cross-Validation for Model Development and Evaluation in Health Care: Tutorial

Cross-validation remains a popular means of developing and validating artificial intelligence for health care. Numerous subtypes of cross-validation exist. Although tutorials on this validation strategy have been published and some with applied examples, we present here a practical tutorial comparing multiple forms of cross-validation using a widely accessible, real-world electronic health care data set: Medical Information Mart for Intensive Care-III (MIMIC-III). This tutorial explored methods such as K-fold cross-validation and nested cross-validation, highlighting their advantages and disadvantages across 2 common predictive modeling use cases: classification (mortality) and regression (length of stay). We aimed to provide readers with reproducible notebooks and best practices for modeling with electronic health care data. We also described sets of useful recommendations as we demonstrated that nested cross-validation reduces optimistic bias but comes with additional computational challenges. This tutorial might improve the community's understanding of these important methods while catalyzing the modeling community to apply these guides directly in their work using the published code.

Evaluating overtaking and filtering maneuver of motorcyclists and car drivers using advanced trajectory data analysis

The present paper compares motorized two-wheeler (MTW) and passenger car's interactions with the rest of the traffic in urban roads while performing overtaking and filtering maneuvers. To better understand filtering maneuvers of motorcyclists and car drivers, an attempt was made to propose a new measure, i.e. pore size ratio. Additionally, the factors affecting lateral width acceptance for motorcyclists and car drivers while overtaking and filtering were studied using advanced trajectory data. A regression model was developed to predict the significant factors affecting motorcyclist's and car driver's decisions to accept lateral width with the adjacent vehicle while performing overtaking and filtering maneuvers. Finally, a comparative analysis between machine learning and the probit model revealed that, in the present case, machine learning models perform better than the probit model in terms of the model's discernment power. The findings of this study will help ameliorate the power of existing microsimulation tools.

Visible and Near-Infrared Spectroscopy Enables Differentiation of Normal and Early Osteoarthritic Human Knee Joint Articular Cartilage

Osteoarthritis degenerates cartilage and impairs joint function. Early intervention opportunities are missed as current diagnostic methods are insensitive to early tissue degeneration. We investigated the capability of visible light-near-infrared spectroscopy (Vis-NIRS) to differentiate normal human cartilage from early osteoarthritic one. Vis-NIRS spectra, biomechanical properties and the state of osteoarthritis (OARSI grade) were quantified from osteochondral samples harvested from different anatomical sites of human cadaver knees. Two support vector machines (SVM) classifiers were developed based on the Vis-NIRS spectra and OARSI scores. The first classifier was designed to distinguish normal (OARSI: 0-1) from general osteoarthritic cartilage (OARSI: 2-5) to check the general suitability of the approach yielding an average accuracy of 75% (AUC = 0.77). Then, the second classifier was designed to distinguish normal from early osteoarthritic cartilage (OARSI: 2-3) yielding an average accuracy of 71% (AUC = 0.73). Important wavelength regions for differentiating normal from early osteoarthritic cartilage were related to collagen organization (wavelength region: 400-600 nm), collagen content (1000-1300 nm) and proteoglycan content (1600-1850 nm). The findings suggest that Vis-NIRS allows objective differentiation of normal and early osteoarthritic tissue, e.g., during arthroscopic repair surgeries.

Narrowing the gap: expected versus deployment performance

OBJECTIVES

Successful model development requires both an accurate a priori understanding of future performance and high performance on deployment. Optimistic estimations of model performance that are unrealized in real-world clinical settings can contribute to nonuse of predictive models. This study used 2 tasks, predicting ICU mortality and Bi-Level Positive Airway Pressure failure, to quantify: (1) how well internal test performances derived from different methods of partitioning data into development and test sets estimate future deployment performance of Recurrent Neural Network models and (2) the effects of including older data in the training set on models' performance.

MATERIALS AND METHODS

The cohort consisted of patients admitted between 2010 and 2020 to the Pediatric Intensive Care Unit of a large quaternary children's hospital. 2010-2018 data were partitioned into different development and test sets to measure internal test performance. Deployable models were trained on 2010-2018 data and assessed on 2019-2020 data, which was conceptualized to represent a real-world deployment scenario. Optimism, defined as the overestimation of the deployed performance by internal test performance, was measured. Performances of deployable models were also compared with each other to quantify the effect of including older data during training.

RESULTS, DISCUSSION, AND CONCLUSION

Longitudinal partitioning methods, where models are tested on newer data than the development set, yielded the least optimism. Including older years in the training dataset did not degrade deployable model performance. Using all available data for model development fully leveraged longitudinal partitioning by measuring year-to-year performance.

Predicting Fetal Alcohol Spectrum Disorders Using Machine Learning Techniques: Multisite Retrospective Cohort Study

BACKGROUND

Fetal alcohol syndrome (FAS) is a lifelong developmental disability that occurs among individuals with prenatal alcohol exposure (PAE). With improved prediction models, FAS can be diagnosed or treated early, if not completely prevented.

OBJECTIVE

In this study, we sought to compare different machine learning algorithms and their FAS predictive performance among women who consumed alcohol during pregnancy. We also aimed to identify which variables (eg, timing of exposure to alcohol during pregnancy and type of alcohol consumed) were most influential in generating an accurate model.

METHODS

Data from the collaborative initiative on fetal alcohol spectrum disorders from 2007 to 2017 were used to gather information about 595 women who consumed alcohol during pregnancy at 5 hospital sites around the United States. To obtain information about PAE, questionnaires or in-person interviews, as well as reviews of medical, legal, or social service records were used to gather information about alcohol consumption. Four different machine learning algorithms (logistic regression, XGBoost, light gradient-boosting machine, and CatBoost) were trained to predict the prevalence of FAS at birth, and model performance was measured by analyzing the area under the receiver operating characteristics curve (AUROC). Of the total cases, 80% were randomly selected for training, while 20% remained as test data sets for predicting FAS. Feature importance was also analyzed using Shapley values for the best-performing algorithm.

RESULTS

Overall, there were 20 cases of FAS within a total population of 595 individuals with PAE. Most of the drinking occurred in the first trimester only (n=491) or throughout all 3 trimesters (n=95); however, there were also reports of drinking in the first and second trimesters only (n=8), and 1 case of drinking in the third trimester only (n=1). The CatBoost method delivered the best performance in terms of AUROC (0.92) and area under the precision-recall curve (AUPRC 0.51), followed by the logistic regression method (AUROC 0.90; AUPRC 0.59), the light gradient-boosting machine (AUROC 0.89; AUPRC 0.52), and XGBoost (AUROC 0.86; AURPC 0.45). Shapley values in the CatBoost model revealed that 12 variables were considered important in FAS prediction, with drinking throughout all 3 trimesters of pregnancy, maternal age, race, and type of alcoholic beverage consumed (eg, beer, wine, or liquor) scoring highly in overall feature importance. For most predictive measures, the best performance was obtained by the CatBoost algorithm, with an AUROC of 0.92, precision of 0.50, specificity of 0.29, F1 score of 0.29, and accuracy of 0.96.

CONCLUSIONS

Machine learning algorithms were able to identify FAS risk with a prediction performance higher than that of previous models among pregnant drinkers. For small training sets, which are common with FAS, boosting mechanisms like CatBoost may help alleviate certain problems associated with data imbalances and difficulties in optimization or generalization.

ECG Recordings as Predictors of Very Early Autism Likelihood: A Machine Learning Approach

In recent years, there has been a rise in the prevalence of autism spectrum disorder (ASD). The diagnosis of ASD requires behavioral observation and standardized testing completed by highly trained experts. Early intervention for ASD can begin as early as 1-2 years of age, but ASD diagnoses are not typically made until ages 2-5 years, thus delaying the start of intervention. There is an urgent need for non-invasive biomarkers to detect ASD in infancy. While previous research using physiological recordings has focused on brain-based biomarkers of ASD, this study investigated the potential of electrocardiogram (ECG) recordings as an ASD biomarker in 3-6-month-old infants. We recorded the heart activity of infants at typical and elevated familial likelihood for ASD during naturalistic interactions with objects and caregivers. After obtaining the ECG signals, features such as heart rate variability (HRV) and sympathetic and parasympathetic activities were extracted. Then we evaluated the effectiveness of multiple machine learning classifiers for classifying ASD likelihood. Our findings support our hypothesis that infant ECG signals contain important information about ASD familial likelihood. Amongthe various machine learning algorithms tested, KNN performed best according to sensitivity (0.70 ± 0.117), F1-score (0.689 ± 0.124), precision (0.717 ± 0.128), accuracy (0.70 ± 0.117, p-value = 0.02), and ROC (0.686 ± 0.122, p-value = 0.06). These results suggest that ECG signals contain relevant information about the likelihood of an infant developing ASD. Future studies should consider the potential of information contained in ECG, and other indices of autonomic control, for the development of biomarkers of ASD in infancy.

The Application of Deep Learning for the Segmentation and Classification of Coronary Arteries

In recent years, the prevalence of coronary artery disease (CAD) has become one of the leading causes of death around the world. Accurate stenosis detection of coronary arteries is crucial for timely treatment. Cardiologists use visual estimations when reading coronary angiography images to diagnose stenosis. As a result, they face various challenges which include high workloads, long processing times and human error. Computer-aided segmentation and classification of coronary arteries, as to whether stenosis is present or not, significantly reduces the workload of cardiologists and human errors caused by manual processes. Moreover, deep learning techniques have been shown to aid medical experts in diagnosing diseases using biomedical imaging. Thus, this study proposes the use of automatic segmentation of coronary arteries using U-Net, ResUNet-a, UNet++, models and classification using DenseNet201, EfficientNet-B0, Mobilenet-v2, ResNet101 and Xception models. In the case of segmentation, the comparative analysis of the three models has shown that U-Net achieved the highest score with a 0.8467 Dice score and 0.7454 Jaccard Index in comparison with UNet++ and ResUnet-a. Evaluation of the classification model's performances has shown that DenseNet201 performed better than other pretrained models with 0.9000 accuracy, 0.9833 specificity, 0.9556 PPV, 0.7746 Cohen's Kappa and 0.9694 Area Under the Curve (AUC).

Transformers for cardiac patient mortality risk prediction from heterogeneous electronic health records

With over 17 million annual deaths, cardiovascular diseases (CVDs) dominate the cause of death statistics. CVDs can deteriorate the quality of life drastically and even cause sudden death, all the while inducing massive healthcare costs. This work studied state-of-the-art deep learning techniques to predict increased risk of death in CVD patients, building on the electronic health records (EHR) of over 23,000 cardiac patients. Taking into account the usefulness of the prediction for chronic disease patients, a prediction period of six months was selected. Two major transformer models that rely on learning bidirectional dependencies in sequential data, BERT and XLNet, were trained and compared. To our knowledge, the presented work is the first to apply XLNet on EHR data to predict mortality. The patient histories were formulated as time series consisting of varying types of clinical events, thus enabling the model to learn increasingly complex temporal dependencies. BERT and XLNet achieved an average area under the receiver operating characteristic curve (AUC) of 75.5% and 76.0%, respectively. XLNet surpassed BERT in recall by 9.8%, suggesting that it captures more positive cases than BERT, which is the main focus of recent research on EHRs and transformers.

Discovery and Validation of Circulating microRNAs as Biomarkers for Epileptogenesis after Experimental Traumatic Brain Injury-The EPITARGET Cohort

Traumatic brain injury (TBI) causes 10-20% of structural epilepsies and 5% of all epilepsies. The lack of prognostic biomarkers for post-traumatic epilepsy (PTE) is a major obstacle to the development of anti-epileptogenic treatments. Previous studies revealed TBI-induced alterations in blood microRNA (miRNA) levels, and patients with epilepsy exhibit dysregulation of blood miRNAs. We hypothesized that acutely altered plasma miRNAs could serve as prognostic biomarkers for brain damage severity and the development of PTE. To investigate this, epileptogenesis was induced in adult male Sprague Dawley rats by lateral fluid-percussion-induced TBI. Epilepsy was defined as the occurrence of at least one unprovoked seizure during continuous 1-month video-electroencephalography monitoring in the sixth post-TBI month. Cortical pathology was analyzed by magnetic resonance imaging on day 2 (D2), D7, and D21, and by histology 6 months post-TBI. Small RNA sequencing was performed from tail-vein plasma samples on D2 and D9 after TBI (n = 16, 7 with and 9 without epilepsy) or sham operation (n = 4). The most promising miRNA biomarker candidates were validated by droplet digital polymerase chain reaction in a validation cohort of 115 rats (8 naïve, 17 sham, and 90 TBI rats [21 with epilepsy]). These included 7 brain-enriched plasma miRNAs (miR-434-3p, miR-9a-3p, miR-136-3p, miR-323-3p, miR-124-3p, miR-212-3p, and miR-132-3p) that were upregulated on D2 post-TBI (p < 0.001 for all compared with naïve rats). The acute post-TBI plasma miRNA profile did not predict the subsequent development of PTE or PTE severity. Plasma miRNA levels, however, predicted the cortical pathology severity on D2 (Spearman ρ = 0.345-0.582, p < 0.001), D9 (ρ = 0.287-0.522, p < 0.001-0.01), D21 (ρ = 0.269-0.581, p < 0.001-0.05) and at 6 months post-TBI (ρ = 0.230-0.433, p < 0.001-0.05). We found that the levels of 6 of 7 miRNAs also reflected mild brain injury caused by the craniotomy during sham operation (ROC AUC 0.76-0.96, p < 0.001-0.05). In conclusion, our findings revealed that increased levels of neuronally enriched miRNAs in the blood circulation after TBI reflect the extent of cortical injury in the brain but do not predict PTE development.

Predicting depressive symptoms in middle-aged and elderly adults using sleep data and clinical health markers: A machine learning approach

OBJECTIVES

Comorbid depression is a highly prevalent and debilitating condition in middle-aged and elderly adults, particularly when associated with obesity, diabetes, and sleep disturbances. In this context, there is a growing need to develop efficient screening methods for cases based on clinical health markers for these comorbidities and sleep data. Thus, our objective was to detect depressive symptoms in these subjects, considering general biomarkers of obesity and diabetes and variables related to sleep and physical exercise through a machine learning approach.

METHODS

We used the National Health and Nutrition Examination Survey (NHANES) 2015-2016 data. Eighteen variables on self-reported physical activity, self-reported sleep habits, sleep disturbance indicative, anthropometric measurements, sociodemographic characteristics and plasma biomarkers of obesity and diabetes were selected as predictors. A total of 2907 middle-aged and elderly subjects were eligible for the study. Supervised learning algorithms such as Lasso penalized Logistic Regression (LR), Random Forest (RF) and Extreme Gradient Boosting (XGBoost) were implemented.

RESULTS

XGBoost provided greater accuracy and precision (87%), with a proportion of hits in cases with depressive symptoms above 80%. In addition, daytime sleepiness was the most significant predictor variable for predicting depressive symptoms.

CONCLUSIONS

Sleep and physical activity variables, in addition to obesity and diabetes biomarkers, together assume significant importance to predict, with accuracy and precision of 87%, the occurrence of depressive symptoms in middle-aged and elderly individuals.

A predictive decision support system for coronavirus disease 2019 response management and medical logistic planning

Objective

Coronavirus disease 2019 demonstrated the inconsistencies in adequately responding to biological threats on a global scale due to a lack of powerful tools for assessing various factors in the formation of the epidemic situation and its forecasting. Decision support systems have a role in overcoming the challenges in health monitoring systems in light of current or future epidemic outbreaks. This paper focuses on some applied examples of logistic planning, a key service of the Earth Cognitive System for Coronavirus Disease 2019 project, here presented, evidencing the added value of artificial intelligence algorithms towards predictive hypotheses in tackling health emergencies.

Methods

Earth Cognitive System for Coronavirus Disease 2019 is a decision support system designed to support healthcare institutions in monitoring, management and forecasting activities through artificial intelligence, social media analytics, geospatial analysis and satellite imaging. The monitoring, management and prediction of medical equipment logistic needs rely on machine learning to predict the regional risk classification colour codes, the emergency rooms attendances, and the forecast of regional medical supplies, synergically enhancing geospatial and temporal dimensions.

Results

The overall performance of the regional risk colour code classifier yielded a high value of the macro-average F1-score (0.82) and an accuracy of 85%. The prediction of the emergency rooms attendances for the Lazio region yielded a very low root mean square error (<11 patients) and a high positive correlation with the actual values for the major hospitals of the Lazio region which admit about 90% of the region's patients. The prediction of the medicinal purchases for the regions of Lazio and Piemonte has yielded a low root mean squared percentage error of 16%.

Conclusions

Accurate forecasting of the evolution of new cases and drug utilisation enables the resulting excess demand throughout the supply chain to be managed more effectively. Forecasting during a pandemic becomes essential for effective government decision-making, managing supply chain resources, and for informing tough policy decisions.

Personalized predictions of adverse side effects of the COVID-19 vaccines

Background

Misconceptions about adverse side effects are thought to influence public acceptance of the Coronavirus disease 2019 (COVID-19) vaccines negatively. To address such perceived disadvantages of vaccines, a novel machine learning (ML) approach was designed to generate personalized predictions of the most common adverse side effects following injection of six different COVID-19 vaccines based on personal and health-related characteristics.

Methods

Prospective data of adverse side effects following COVID-19 vaccination in 19943 participants from Iran and Switzerland was utilized. Six vaccines were studied: The AZD1222, Sputnik V, BBIBP-CorV, COVAXIN, BNT162b2, and the mRNA-1273 vaccine. The eight side effects were considered as the model output: fever, fatigue, headache, nausea, chills, joint pain, muscle pain, and injection site reactions. The total input parameters for the first and second dose predictions were 46 and 54 features, respectively, including age, gender, lifestyle variables, and medical history. The performances of multiple ML models were compared using Area Under the Receiver Operating Characteristic Curve (ROC-AUC).

Results

The total number of people receiving the first dose of the AZD1222, Sputnik V, BBIBP-CorV, COVAXIN, BNT162b2, and mRNA-1273 were 6022, 7290, 5279, 802, 277, and 273, respectively. For the second dose, the numbers were 2851, 5587, 3841, 599, 242 and 228. The Logistic Regression model for predicting different side effects of the first dose achieved ROC-AUCs of 0.620-0.686, 0.685-0.716, 0.632-0.727, 0.527-0.598, 0.548-0.655, 0.545-0.712 for the AZD1222, Sputnik V, BBIBP-CorV, COVAXIN, BNT162b2 and mRNA-1273 vaccines, respectively. The second dose models yielded ROC-AUCs of 0.777-0.867, 0.795-0.848, 0.857-0.906, 0.788-0.875, 0.683-0.850, and 0.486-0.680, respectively.

Conclusions

Using a large cohort of recipients vaccinated with COVID-19 vaccines, a novel and personalized strategy was established to predict the occurrence of the most common adverse side effects with high accuracy. This technique can serve as a tool to inform COVID-19 vaccine selection and generate personalized factsheets to curb concerns about adverse side effects.

Evaluating artificial intelligence algorithms for use in veterinary radiology

Artificial intelligence is increasingly being used for applications in veterinary radiology, including detection of abnormalities and automated measurements. Unlike human radiology, there is no formal regulation or validation of AI algorithms for veterinary medicine and both general practitioner and specialist veterinarians must rely on their own judgment when deciding whether or not to incorporate AI algorithms to aid their clinical decision-making. The benefits and challenges to developing clinically useful and diagnostically accurate AI algorithms are discussed. Considerations for the development of AI research projects are also addressed. A framework is suggested to help veterinarians, in both research and clinical practice contexts, assess AI algorithms for veterinary radiology.

Eye movement behavior in a real-world virtual reality task reveals ADHD in children

Eye movements and other rich data obtained in virtual reality (VR) environments resembling situations where symptoms are manifested could help in the objective detection of various symptoms in clinical conditions. In the present study, 37 children with attention deficit hyperactivity disorder and 36 typically developing controls (9-13 y.o) played a lifelike prospective memory game using head-mounted display with inbuilt 90 Hz eye tracker. Eye movement patterns had prominent group differences, but they were dispersed across the full performance time rather than associated with specific events or stimulus features. A support vector machine classifier trained on eye movement data showed excellent discrimination ability with 0.92 area under curve, which was significantly higher than for task performance measures or for eye movements obtained in a visual search task. We demonstrated that a naturalistic VR task combined with eye tracking allows accurate prediction of attention deficits, paving the way for precision diagnostics.

Comparison of conventional scoring systems to machine learning models for the prediction of major adverse cardiovascular events in patients undergoing coronary computed tomography angiography

BACKGROUND

The study aims to compare the prognostic performance of conventional scoring systems to a machine learning (ML) model on coronary computed tomography angiography (CCTA) to discriminate between the patients with and without major adverse cardiovascular events (MACEs) and to find the most important contributing factor of MACE.

MATERIALS AND METHODS

From November to December 2019, 500 of 1586 CCTA scans were included and analyzed, then six conventional scores were calculated for each participant, and seven ML models were designed. Our study endpoints were all-cause mortality, non-fatal myocardial infarction, late coronary revascularization, and hospitalization for unstable angina or heart failure. Score performance was assessed by area under the curve (AUC) analysis.

RESULTS

Of 500 patients (mean age: 60 ± 10; 53.8% male subjects) referred for CCTA, 416 patients have met inclusion criteria, 46 patients with early (<90 days) cardiac evaluation (due to the inability to clarify the reason for the assessment, deterioration of the symptoms vs. the CCTA result), and 38 patients because of missed follow-up were not enrolled in the final analysis. Forty-six patients (11.0%) developed MACE within 20.5 ± 7.9 months of follow-up. Compared to conventional scores, ML models showed better performance, except only one model which is eXtreme Gradient Boosting had lower performance than conventional scoring systems (AUC:0.824, 95% confidence interval (CI): 0.701-0.947). Between ML models, random forest, ensemble with generalized linear, and ensemble with naive Bayes were shown to have higher prognostic performance (AUC: 0.92, 95% CI: 0.85-0.99, AUC: 0.90, 95% CI: 0.81-0.98, and AUC: 0.89, 95% CI: 0.82-0.97), respectively. Coronary artery calcium score (CACS) had the highest correlation with MACE.

CONCLUSION

Compared to the conventional scoring system, ML models using CCTA scans show improved prognostic prediction for MACE. Anatomical features were more important than clinical characteristics.

Constructing a molecular subtype model of colon cancer using machine learning

Background: Colon cancer (CRC) is one of the malignant tumors with a high incidence in the world. Many previous studies on CRC have focused on clinical research. With the in-depth study of CRC, the role of molecular mechanisms in CRC has become increasingly important. Currently, machine learning is widely used in medicine. By combining machine learning with molecular mechanisms, we can better understand CRC’s pathogenesis and develop new treatments for it.

Methods and materials: We used the R language to construct molecular subtypes of colon cancer and subsequently explored prognostic genes with GEPIA2. Enrichment analysis is used by WebGestalt to obtain differential genes. Protein–protein interaction networks of differential genes were constructed using the STRING database and the Cytoscape tool. TIMER2.0 and TISIDB databases were used to investigate the correlation of these genes with immune-infiltrating cells and immune targets. The cBioportal database was used to explore genomic alterations.

Results: In our study, the molecular prognostic model of CRC was constructed to study the prognostic factors of CRC, and finally, it was found that Charcot–Leyden crystal galectin (CLC), zymogen granule protein 16 (ZG16), leucine-rich repeat-containing protein 26 (LRRC26), intelectin 1 (ITLN1), UDP-GlcNAc: betaGal beta-1,3-N-acetylglucosaminyltransferase 6 (B3GNT6), chloride channel accessory 1 (CLCA1), growth factor independent 1 transcriptional repressor (GFI1), aquaporin 8 (AQP8), HEPACAM family member 2 (HEPACAM2), and UDP glucuronosyltransferase family 2 member B15 (UGT2B15) were correlated with the subtype model of CRC prognosis. Enrichment analysis shows that differential genes were mainly associated with immune-inflammatory pathways. GFI1 and CLC were associated with immune cells, immunoinhibitors, and immunostimulator. Genomic analysis shows that there were no significant changes in differential genes.

Conclusion: By constructing molecular subtypes of colon cancer, we discovered new colon cancer prognostic markers, which can provide direction for new treatments in the future.

Predicting Patient-Level 3-Level Version of EQ-5D Index Scores From a Large International Database Using Machine Learning and Regression Methods

OBJECTIVES

This study aimed to evaluate the performance of machine learning and regression methods in the prediction of 3-level version of EQ-5D (EQ-5D-3L) index scores from a large diverse data set.

METHODS

A total of 30 studies from 3 countries were combined. Predictions were performed via eXtreme Gradient Boosting classification (XGBC), eXtreme Gradient Boosting regression (XGBR) and ordinary least squares (OLS) regression using 10-fold cross-validation and 80%/20% partition for training and testing. We evaluated 6 prediction scenarios using 3 samples (general population, patients, total) and 2 predictor sets: demographic and disease-related variables with/without patient-reported outcomes. Model performance was evaluated by mean absolute error and percent of predictions within clinically irrelevant error range and within correct health severity group (EQ-5D-3L index <0.45, 0.45-0.926, >0.926).

RESULTS

The data set involved 26 318 individuals (clinical settings n = 6214, general population n = 20 104) and 26 predictor variables plus diagnoses. Using all predictors and the total sample, mean absolute error values were 0.153, 0.126, and 0.131, percent of predictions within clinically irrelevant error range were 47.6%, 39.5%, and 37.4%, and within the correct health severity group were 56.3%, 64.9%, and 63.3% by XGBC, XGBR, and OLS, respectively. The performance of models depended on the applied evaluation criteria, the target population, the included predictors, and the EQ-5D-3L index score range.

CONCLUSIONS

Regression models (XGBR and OLS) outperformed XGBC, yet prediction errors were outside the clinically irrelevant error range for most respondents. Our results highlight the importance of systematic patient-reported outcome (EQ-5D) data collection. Dialogs between artificial intelligence and outcomes research experts are encouraged to enhance the value of accumulating data in health systems.

Analysis of the feasibility of using deep learning for multiclass classification of dental anomalies on panoramic radiographs

The aim of the feasibility study was to construct deep learning models for the classification of multiple dental anomalies in panoramic radiographs. Panoramic radiographs with single supernumerary teeth and/or odontomas were considered the "case" group; panoramic radiographs with no dental anomalies were considered the "control" group. The dataset comprised 150 panoramic radiographs: 50 each of no dental anomalies, single supernumerary teeth, and odontomas. To classify the panoramic radiographs into case and control categories, we employed AlexNet, which is a convolutional neural network model. AlexNet was able to classify whole panoramic radiographs into two or three classes, according to the presence or absence of supernumerary teeth or odontomas. The performance metrics of the three-class classification were 70%, 70.8%, 70%, and 69.7% for accuracy, precision, sensitivity, and F1 score, respectively, in the macro average. These results support the feasibility of using deep learning to detect multiple dental anomalies in panoramic radiographs.

Intelligent Palliative Care Based on Patient-Reported Outcome Measures

CONTEXT

The growth of patient reported outcome measures data in palliative care provides an opportunity for machine learning to identify patterns in patient responses signifying different phases of illness.

OBJECTIVES

The study will explore if machine learning and network analysis can identify phases in patient palliative status through symptoms reported on the Integrated Palliative Care Outcome Scale (IPOS).

METHODS

A partly cross-sectional and partially longitudinal observational study was undertaken using the Australasian Karnofsky Performance Scale (AKPS); Integrated Palliative Care Outcome Scale (IPOS); Phase of Illness (POI). Patient palliative records (n = 1507, 65% stable, 20% unstable, 9% deteriorating, 2% terminal) from 804 adult patients enrolled in a New Zealand palliative care service were analysed using a combination of statistical, machine learning and network analysis techniques.

RESULTS

Data from IPOS showed considerable variation with phase. Also, network analysis showed clear associations between items by phase. Six machine learning techniques identified the most important variables for predicting possible transition between phases of illness. Network analysis for all patients showed that Poor Appetite and Loss of Energy were central IPOS items, with Loss of Energy linked to Drowsiness, Shortness of Breath and Lack of Mobility on the one hand, and Poor Appetite linked to Nausea, Vomiting, Constipation and Sore and Dry Mouth on the other.

CONCLUSION

These preliminary results, when coupled with the latest technological developments in mobile apps and wearable technology, could point the way to increased use of digital therapeutics in continuous palliative care monitoring.

DeepMC-iNABP: Deep learning for multiclass identification and classification of nucleic acid-binding proteins

Nucleic acid-binding proteins (NABPs), including DNA-binding proteins (DBPs) and RNA-binding proteins (RBPs), play vital roles in gene expression. Accurate identification of these proteins is crucial. However, there are two existing challenges: one is the problem of ignoring DNA- and RNA-binding proteins (DRBPs), and the other is a cross-predicting problem referring to DBP predictors predicting DBPs as RBPs, and vice versa. In this study, we proposed a computational predictor, called DeepMC-iNABP, with the goal of solving these difficulties by utilizing a multiclass classification strategy and deep learning approaches. DBPs, RBPs, DRBPs and non-NABPs as separate classes of data were used for training the DeepMC-iNABP model. The results on test data collected in this study and two independent test datasets showed that DeepMC-iNABP has a strong advantage in identifying the DRBPs and has the ability to alleviate the cross-prediction problem to a certain extent. The web-server of DeepMC-iNABP is freely available at http://www.deepmc-inabp.net/. The datasets used in this research can also be downloaded from the website.

Supporting peace negotiations in the Yemen war through machine learning

Today’s conflicts are becoming increasingly complex, fluid, and fragmented, often involving a host of national and international actors with multiple and often divergent interests. This development poses significant challenges for conflict mediation, as mediators struggle to make sense of conflict dynamics, such as the range of conflict parties and the evolution of their political positions, the distinction between relevant and less relevant actors in peace-making, or the identification of key conflict issues and their interdependence. International peace efforts appear ill-equipped to successfully address these challenges. While technology is already being experimented with and used in a range of conflict related fields, such as conflict predicting or information gathering, less attention has been given to how technology can contribute to conflict mediation. This case study contributes to emerging research on the use of state-of-the-art machine learning technologies and techniques in conflict mediation processes. Using dialogue transcripts from peace negotiations in Yemen, this study shows how machine-learning can effectively support mediating teams by providing them with tools for knowledge management, extraction and conflict analysis. Apart from illustrating the potential of machine learning tools in conflict mediation, the article also emphasizes the importance of interdisciplinary and participatory, cocreation methodology for the development of context-sensitive and targeted tools and to ensure meaningful and responsible implementation.

Machine learning in the prediction of depression treatment outcomes: a systematic review and meta-analysis

BACKGROUND

Multiple treatments are effective for major depressive disorder (MDD), but the outcomes of each treatment vary broadly among individuals. Accurate prediction of outcomes is needed to help select a treatment that is likely to work for a given person. We aim to examine the performance of machine learning methods in delivering replicable predictions of treatment outcomes.

METHODS

Of 7732 non-duplicate records identified through literature search, we retained 59 eligible reports and extracted data on sample, treatment, predictors, machine learning method, and treatment outcome prediction. A minimum sample size of 100 and an adequate validation method were used to identify adequate-quality studies. The effects of study features on prediction accuracy were tested with mixed-effects models. Fifty-four of the studies provided accuracy estimates or other estimates that allowed calculation of balanced accuracy of predicting outcomes of treatment.

RESULTS

Eight adequate-quality studies reported a mean accuracy of 0.63 [95% confidence interval (CI) 0.56-0.71], which was significantly lower than a mean accuracy of 0.75 (95% CI 0.72-0.78) in the other 46 studies. Among the adequate-quality studies, accuracies were higher when predicting treatment resistance (0.69) and lower when predicting remission (0.60) or response (0.56). The choice of machine learning method, feature selection, and the ratio of features to individuals were not associated with reported accuracy.

CONCLUSIONS

The negative relationship between study quality and prediction accuracy, combined with a lack of independent replication, invites caution when evaluating the potential of machine learning applications for personalizing the treatment of depression.

Human, All Too Human? An All-Around Appraisal of the "Artificial Intelligence Revolution" in Medical Imaging

Artificial intelligence (AI) has seen dramatic growth over the past decade, evolving from a niche super specialty computer application into a powerful tool which has revolutionized many areas of our professional and daily lives, and the potential of which seems to be still largely untapped. The field of medicine and medical imaging, as one of its various specialties, has gained considerable benefit from AI, including improved diagnostic accuracy and the possibility of predicting individual patient outcomes and options of more personalized treatment. It should be noted that this process can actively support the ongoing development of advanced, highly specific treatment strategies (e.g., target therapies for cancer patients) while enabling faster workflow and more efficient use of healthcare resources. The potential advantages of AI over conventional methods have made it attractive for physicians and other healthcare stakeholders, raising much interest in both the research and the industry communities. However, the fast development of AI has unveiled its potential for disrupting the work of healthcare professionals, spawning concerns among radiologists that, in the future, AI may outperform them, thus damaging their reputations or putting their jobs at risk. Furthermore, this development has raised relevant psychological, ethical, and medico-legal issues which need to be addressed for AI to be considered fully capable of patient management. The aim of this review is to provide a brief, hopefully exhaustive, overview of the state of the art of AI systems regarding medical imaging, with a special focus on how AI and the entire healthcare environment should be prepared to accomplish the goal of a more advanced human-centered world.