Challenges of machine learning model validation using correlated behaviour data: Evaluation of cross-validation strategies and accuracy measures

Classification of sex-dependent specific behaviours by tri-axial acceleration in the tegu lizard Salvator merianae

Validated patterns of behaviour detected by tri-axial acceleration in the laboratory can be used for remote measurements of free-living animals. The tegu lizard naturally occupies diverse biomes in South America and presents ecological threats in regions where it was artificially introduced. We aimed to validate the use of tri-axial acceleration to distinguish among behaviours of male and female tegus in captivity by comparing observed behaviours to recorded acceleration data. Adult animals were externally fitted with an accelerometer fixed between their scapulae to quantify anteroposterior, lateral, and dorsoventral acceleration. Video recordings of cameras positioned on the walls of the animal-holding arena documented behaviours. Behaviour patterns, such as resting, walking, and eating, were identified for both sexes, and nest building in females and courtship and copulation in males. Random Forest algorithm was used to validate the behaviour patterns from accelerometry data based on two models, random split (70 % training-30 % validation; RS) and leave-one-out (divided by individual; LOO). Although LOO showed lower accuracies than RS for all the acceleration data, nest building in females and copulation in males had high accuracies in both models. In contrast, the lowest accuracies for walking and eating indicates they may involve more inconsistent movement patterns. Comparing the results from RS and LOO, female behaviours may be more identifiable in the field using triaxial accelerometry than males. The identification of behaviours by accelerometry, especially related to reproduction, without the necessity for direct observation of the tegus would be helpful for conservation purposes, for both natural and invasive populations.

Unsupervised Domain Adaptation for Mitigating Sensor Variability and Interspecies Heterogeneity in Animal Activity Recognition

Animal activity recognition (AAR) using wearable sensor data has gained significant attention due to its applications in monitoring and understanding animal behavior. However, two major challenges hinder the development of robust AAR models: domain variability and the difficulty of obtaining labeled datasets. To address this issue, this study intensively investigates the impact of unsupervised domain adaptation (UDA) for AAR. We compared three distinct types of UDA techniques: minimizing divergence-based, adversarial-based, and reconstruction-based approaches. By leveraging UDA, AAR classifiers enable the model to learn domain-invariant features, allowing classifiers trained on the source domain to perform well on the target domain without labels. We evaluated the effectiveness of UDA techniques using dog movement sensor data and additional data from horses. The application of UDA across sensor positions (neck and back), sizes (middle-sized and large-sized), and gender (female and male) within the dog data, as well as across species (dog and horses), exhibits significant improvements in the classification performance and reduced the domain discrepancy. The results highlight the potential of UDA to mitigate the domain shift and enhance AAR in various settings and for different animal species, providing valuable insights for practical applications in real-world scenarios where labeled data is scarce.

Animal behaviour on the move: the use of auxiliary information and semi-supervision to improve behavioural inferences from Hidden Markov Models applied to GPS tracking datasets

BACKGROUND

State-space models, such as Hidden Markov Models (HMMs), are increasingly used to classify animal tracks into behavioural states. Typically, step length and turning angles of successive locations are used to infer where and when an animal is resting, foraging, or travelling. However, the accuracy of behavioural classifications is seldom validated, which may badly contaminate posterior analyses. In general, models appear to efficiently infer behaviour in species with discrete foraging and travelling areas, but classification is challenging for species foraging opportunistically across homogenous environments, such as tropical seas. Here, we use a subset of GPS loggers deployed simultaneously with wet-dry data from geolocators, activity measurements from accelerometers, and dive events from Time Depth Recorders (TDR), to improve the classification of HMMs of a large GPS tracking dataset (478 deployments) of red-billed tropicbirds (Phaethon aethereus), a poorly studied pantropical seabird.

METHODS

We classified a subset of fixes as either resting, foraging or travelling based on the three auxiliary sensors and evaluated the increase in overall accuracy, sensitivity (true positive rate), specificity (true negative rate) and precision (positive predictive value) of the models in relation to the increasing inclusion of fixes with known behaviours.

RESULTS

We demonstrate that even with a small informed sub-dataset (representing only 9% of the full dataset), we can significantly improve the overall behavioural classification of these models, increasing model accuracy from 0.77 ± 0.01 to 0.85 ± 0.01 (mean ± sd). Despite overall improvements, the sensitivity and precision of foraging behaviour remained low (reaching 0.37 ± 0.06, and 0.06 ± 0.01, respectively).

CONCLUSIONS

This study demonstrates that the use of a small subset of auxiliary data with known behaviours can both validate and notably improve behavioural classifications of state space models of opportunistic foragers. However, the improvement is state-dependant and caution should be taken when interpreting inferences of foraging behaviour from GPS data in species foraging on the go across homogenous environments.

Machine learning based canine posture estimation using inertial data

The aim of this study was to design a new canine posture estimation system specifically for working dogs. The system was composed of Inertial Measurement Units (IMUs) that are commercially available, and a supervised learning algorithm which was developed for different behaviours. Three IMUs, each containing a 3-axis accelerometer, gyroscope, and magnetometer, were attached to the dogs' chest, back, and neck. To build and test the model, data were collected during a video-recorded behaviour test where the trainee assistance dogs performed static postures (standing, sitting, lying down) and dynamic activities (walking, body shake). Advanced feature extraction techniques were employed for the first time in this field, including statistical, temporal, and spectral methods. The most important features for posture prediction were chosen using Select K Best with ANOVA F-value. The individual contributions of each IMU, sensor, and feature type were analysed using Select K Best scores and Random Forest feature importance. Results showed that the back and chest IMUs were more important than the neck IMU, and the accelerometers were more important than the gyroscopes. The addition of IMUs to the chest and back of dog harnesses is recommended to improve performance. Additionally, statistical and temporal feature domains were more important than spectral feature domains. Three novel cascade arrangements of Random Forest and Isolation Forest were fitted to the dataset. The best classifier achieved an f1-macro of 0.83 and an f1-weighted of 0.90 for the prediction of the five postures, demonstrating a better performance than previous studies. These results were attributed to the data collection methodology (number of subjects and observations, multiple IMUs, use of common working dog breeds) and novel machine learning techniques (advanced feature extraction, feature selection and modelling arrangements) employed. The dataset and code used are publicly available on Mendeley Data and GitHub, respectively.

Hybrid BO-XGBoost and BO-RF Models for the Strength Prediction of Self-Compacting Mortars with Parametric Analysis

Self-compacting mortar (SCM) has superior workability and long-term durable performance compared to traditional mortar. The strength of SCM, including both its compressive and flexural strengths, is a crucial property that is determined by appropriate curing conditions and mix design parameters. In the context of materials science, predicting the strength of SCM is challenging because of multiple influencing factors. This study employed machine learning techniques to establish SCM strength prediction models. Based on ten different input parameters, the strength of SCM specimens were predicted using two different types of hybrid machine learning (HML) models, namely Extreme Gradient Boosting (XGBoost) and the Random Forest (RF) algorithm. HML models were trained and tested by experimental data from 320 test specimens. In addition, the Bayesian optimization method was utilized to fine tune the hyperparameters of the employed algorithms, and cross-validation was employed to partition the database into multiple folds for a more thorough exploration of the hyperparameter space while providing a more accurate assessment of the model's predictive power. The results show that both HML models can successfully predict the SCM strength values with high accuracy, and the Bo-XGB model demonstrated higher accuracy (R² = 0.96 for training and R² = 0.91 for testing phases) for predicting flexural strength with low error. In terms of compressive strength prediction, the employed BO-RF model performed very well, with R² = 0.96 for train and R² = 0.88 testing stages with minor errors. Moreover, the SHAP algorithm, permutation importance and leave-one-out importance score were used for sensitivity analysis to explain the prediction process and interpret the governing input variable parameters of the proposed HML models. Finally, the outcomes of this study might be applied to guide the future mix design of SCM specimens.

Application of machine learning in the prediction of deficient mismatch repair in patients with colorectal cancer based on routine preoperative characterization

Simple summary

Detecting deficient mismatch repair (dMMR) in patients with colorectal cancer is essential for clinical decision-making, including evaluation of prognosis, guidance of adjuvant chemotherapy and immunotherapy, and primary screening for Lynch syndrome. However, outside of tertiary care centers, existing detection methods are not widely disseminated and highly depend on the experienced pathologist. Therefore, it is of great clinical significance to develop a broadly accessible and low-cost tool for dMMR prediction, particularly prior to surgery. In this study, we developed a convenient and reliable model for predicting dMMR status in CRC patients on routine preoperative characterization utilizing multiple machine learning algorithms. This model will work as an automated screening tool for identifying patients suitable for mismatch repair testing and consequently for improving the detection rate of dMMR, while reducing unnecessary labor and cost in patients with proficient mismatch repair.

Background

Deficient mismatch repair (dMMR) indicates a sustained anti-tumor immune response and has a favorable prognosis in patients with colorectal cancer (CRC). Although all CRC patients are recommended to undergo dMMR testing after surgery, current diagnostic approaches are not available for all country hospitals and patients. Therefore, efficient and low-cost predictive models for dMMR, especially for preoperative evaluations, are warranted.

Methods

A large scale of 5596 CRC patients who underwent surgical resection and mismatch repair testing were enrolled and randomly divided into training and validation cohorts. The clinical features exploited for predicting dMMR comprised the demographic characteristics, preoperative laboratory data, and tumor burden information. Machine learning (ML) methods involving eight basic algorithms, ensemble learning methods, and fusion algorithms were adopted with 10-fold cross-validation, and their performance was evaluated based on the area under the receiver operating characteristic curve (AUC) and calibration curves. The clinical net benefits were assessed using a decision curve analysis (DCA), and a nomogram was developed to facilitate model clinical practicality.

Results

All models achieved an AUC of nearly 0.80 in the validation cohort, with the stacking model exhibiting the best performance (AUC = 0.832). Logistical DCA revealed that the stacking model yielded more clinical net benefits than the conventional regression models. In the subgroup analysis, the stacking model also predicted dMMR regardless of the clinical stage. The nomogram showed a favorable consistence with the actual outcome in the calibration curve.

Conclusion

With the aid of ML algorithms, we developed a novel and robust model for predicting dMMR in CRC patients with satisfactory discriminative performance and designed a user-friendly and convenient nomogram.

Enhanced Classification of Dog Activities with Quaternion-Based Fusion Approach on High-Dimensional Raw Data from Wearable Sensors

The employment of machine learning algorithms to the data provided by wearable movement sensors is one of the most common methods to detect pets' behaviors and monitor their well-being. However, defining features that lead to highly accurate behavior classification is quite challenging. To address this problem, in this study we aim to classify six main dog activities (standing, walking, running, sitting, lying down, and resting) using high-dimensional sensor raw data. Data were received from the accelerometer and gyroscope sensors that are designed to be attached to the dog's smart costume. Once data are received, the module computes a quaternion value for each data point that provides handful features for classification. Next, to perform the classification, we used several supervised machine learning algorithms, such as the Gaussian naïve Bayes (GNB), Decision Tree (DT), K-nearest neighbor (KNN), and support vector machine (SVM). In order to evaluate the performance, we finally compared the proposed approach's F-score accuracies with the accuracy of classic approach performance, where sensors' data are collected without computing the quaternion value and directly utilized by the model. Overall, 18 dogs equipped with harnesses participated in the experiment. The results of the experiment show a significantly enhanced classification with the proposed approach. Among all the classifiers, the GNB classification model achieved the highest accuracy for dog behavior. The behaviors are classified with F-score accuracies of 0.94, 0.86, 0.94, 0.89, 0.95, and 1, respectively. Moreover, it has been observed that the GNB classifier achieved 93% accuracy on average with the dataset consisting of quaternion values. In contrast, it was only 88% when the model used the dataset from sensors' data.

Developing a classification system to assign activity states to two species of freshwater turtles

Research in ecology often requires robust assessment of animal behaviour, but classifying behavioural patterns in free-ranging animals and in natural environments can be especially challenging. New miniaturised bio-logging devices such as accelerometers are increasingly available to record animal behaviour remotely, and thereby address the gap in knowledge related to behaviour of free-ranging animals. However, validation of these data is rarely conducted and classification model transferability across closely-related species is often not tested. Here, we validated accelerometer and water sensor data to classify activity states in two free-ranging freshwater turtle species (Blanding's turtle, Emydoidea blandingii, and Painted turtle, Chrysemys picta). First, using only accelerometer data, we developed a decision tree to separate motion from motionless states, and second, we included water sensor data to classify the animal as being motionless or in-motion on land or in water. We found that accelerometers separated in-motion from motionless behaviour with > 83% accuracy, whereas models also including water sensor data predicted states in terrestrial and aquatic locations with > 77% accuracy. Despite differences in values separating activity states between the two species, we found high model transferability allowing cross-species application of classification models. Note that reducing sampling frequency did not affect predictive accuracy of our models up to a sampling frequency of 0.0625 Hz. We conclude that the use of accelerometers in animal research is promising, but requires prior data validation and development of robust classification models, and whenever possible cross-species assessment should be conducted to establish model generalisability.

Forecasting determinants of recurrence in lung cancer patients exploiting various machine learning models

Lung cancer recurrence seems to be the most leading cause of death as well as deterioration of lifespan. Proper assessment of the probability of recurrence in early-stage lung cancer is necessary to push up the treatment progress. We therefore employed machine-learning technologies to forecast post-operative recurrence risks using 174 lung cancer patient records. Six classification algorithms logistic regression, SVM, decision tree classification, random forest classification, XGBoost and lightGBM were used to predict the cancer recurrence. The patient samples were divided into training and test group with the split ratio of 3:1 for model generation and the accuracy were validated using k-fold cross-validation method. It is worth noting that the logistic regression model outperformed all the models in both training (Accuracy = 0.82) and test set (Accuracy = 0.79) on k-fold validation. Further, the optimal features (n = 7) identified using the RFE method is certainly helpful to improve the model in a high precision. The imperative risk factors associated with recurrence were identified using three feature selection methods. Importantly, our research showed that age is an important prognostic factor to be considered during the recurrence prediction. Indeed, severe concern on the identified risk factors combined with predictive models assists the physician to reduce the cancer recurrence rate in patients with lung cancer.

Human activity recognition of children with wearable devices using LightGBM machine learning

Human activity recognition (HAR) using machine learning (ML) methods has been a continuously developed method for collecting and analyzing large amounts of human behavioral data using special wearable sensors in the past decade. Our main goal was to find a reliable method that could automatically detect various playful and daily routine activities in children. We defined 40 activities for ML recognition, and we collected activity motion data by means of wearable smartwatches with a special SensKid software. We analyzed the data of 34 children (19 girls, 15 boys; age range: 6.59–8.38; median age = 7.47). All children were typically developing first graders from three elementary schools. The activity recognition was a binary classification task which was evaluated with a Light Gradient Boosted Machine (LGBM) learning algorithm, a decision tree based method with a threefold cross validation. We used the sliding window technique during the signal processing, and we aimed at finding the best window size for the analysis of each behavior element to achieve the most effective settings. Seventeen activities out of 40 were successfully recognized with AUC values above 0.8. The window size had no significant effect. In summary, the LGBM is a very promising solution for HAR. In line with previous findings, our results provide a firm basis for a more precise and effective recognition system that can make human behavioral analysis faster and more objective.

Uncertainty in experts' judgments exposes the vulnerability of research reporting anecdotes on animals' cognitive abilities

Expertise in science, particularly in animal behaviour, may provide people with the capacity to provide better judgments in contrast to lay people. Here we explore whether experts provide a more objective, accurate and coherent evaluation of a recently reported anecdote on Atlantic puffin (Fratercula arctica) “tool use” (recorded on video) which was published in a major scientific journal but was received with some scepticism. We relied on citizen science and developed a questionnaire to measure whether experts in ethology and ornithology and lay people agree or disagree on (1) the description of the actions that they observe (the bird takes a stick in its beak), (2) the possible goal of the action (nest-building or grooming) and (3) the intentional component of the action (the bird took the stick into its beak in order to scratch itself). We hypothesised that contrary to the lay people, experts are more critical evaluators that is they are more inclined to report alternative actions, like nest building, or are less likely to attributing goal-directedness to the action in the absence of evidence. In contrast, lay people may be more prone to anthropomorphise utilising a teleological and intentional stance. Alternatively, all three groups of subjects may rely on anthropomorphism at similar levels and prior expertise does not play a significant role. We found that no major differences among the evaluators. At the group levels, respondents were relatively uncertain with regard to the action of the bird seen on the video but they showed some individual consistency with regard to the description of the action. Thus, we conclude that paradoxically, with regard to the task our experts are typically not experts in the strict sense of the definition, and suggest that anecdotal reports should not be used to argue about mental processes.

Limitations of using surrogates for behaviour classification of accelerometer data: refining methods using random forest models in Caprids

BACKGROUND

Animal-attached devices can be used on cryptic species to measure their movement and behaviour, enabling unprecedented insights into fundamental aspects of animal ecology and behaviour. However, direct observations of subjects are often still necessary to translate biologging data accurately into meaningful behaviours. As many elusive species cannot easily be observed in the wild, captive or domestic surrogates are typically used to calibrate data from devices. However, the utility of this approach remains equivocal.

METHODS

Here, we assess the validity of using captive conspecifics, and phylogenetically-similar domesticated counterparts (surrogate species) for calibrating behaviour classification. Tri-axial accelerometers and tri-axial magnetometers were used with behavioural observations to build random forest models to predict the behaviours. We applied these methods using captive Alpine ibex (Capra ibex) and a domestic counterpart, pygmy goats (Capra aegagrus hircus), to predict the behaviour including terrain slope for locomotion behaviours of captive Alpine ibex.

RESULTS

Behavioural classification of captive Alpine ibex and domestic pygmy goats was highly accurate (> 98%). Model performance was reduced when using data split per individual, i.e., classifying behaviour of individuals not used to train models (mean ± sd = 56.1 ± 11%). Behavioural classifications using domestic counterparts, i.e., pygmy goat observations to predict ibex behaviour, however, were not sufficient to predict all behaviours of a phylogenetically similar species accurately (> 55%).

CONCLUSIONS

We demonstrate methods to refine the use of random forest models to classify behaviours of both captive and free-living animal species. We suggest there are two main reasons for reduced accuracy when using a domestic counterpart to predict the behaviour of a wild species in captivity; domestication leading to morphological differences and the terrain of the environment in which the animals were observed. We also identify limitations when behaviour is predicted in individuals that are not used to train models. Our results demonstrate that biologging device calibration needs to be conducted using: (i) with similar conspecifics, and (ii) in an area where they can perform behaviours on terrain that reflects that of species in the wild.

Hidden Markov models identify major movement modes in accelerometer and magnetometer data from four albatross species

BACKGROUND

Inertial measurement units (IMUs) with high-resolution sensors such as accelerometers are now used extensively to study fine-scale behavior in a wide range of marine and terrestrial animals. Robust and practical methods are required for the computationally-demanding analysis of the resulting large datasets, particularly for automating classification routines that construct behavioral time series and time-activity budgets. Magnetometers are used increasingly to study behavior, but it is not clear how these sensors contribute to the accuracy of behavioral classification methods. Development of effective  classification methodology is key to understanding energetic and life-history implications of foraging and other behaviors.

METHODS

We deployed accelerometers and magnetometers on four species of free-ranging albatrosses and evaluated the ability of unsupervised hidden Markov models (HMMs) to identify three major modalities in their behavior: 'flapping flight', 'soaring flight', and 'on-water'. The relative contribution of each sensor to classification accuracy was measured by comparing HMM-inferred states with expert classifications identified from stereotypic patterns observed in sensor data.

RESULTS

HMMs provided a flexible and easily interpretable means of classifying behavior from sensor data. Model accuracy was high overall (92%), but varied across behavioral states (87.6, 93.1 and 91.7% for 'flapping flight', 'soaring flight' and 'on-water', respectively). Models built on accelerometer data alone were as accurate as those that also included magnetometer data; however, the latter were useful for investigating slow and periodic behaviors such as dynamic soaring at a fine scale.

CONCLUSIONS

The use of IMUs in behavioral studies produces large data sets, necessitating the development of computationally-efficient methods to automate behavioral classification in order to synthesize and interpret underlying patterns. HMMs provide an accessible and robust framework for analyzing complex IMU datasets and comparing behavioral variation among taxa across habitats, time and space.

Exploratory study of introducing HPC to non-ICT researchers: institutional strategy is possibly needed for widespread adaption

Machine learning algorithms are becoming more and more useful in many fields of science, including many areas where computational methods are rarely used. High-performance Computing (HPC) is the most powerful solution to get the best results using these algorithms. HPC requires various skills to use. Acquiring this knowledge might be intimidating and take a long time for a researcher with small or no background in information and communications technologies (ICTs), even if the benefits of such knowledge is evident for the researcher. In this work, we aim to assess how a specific method of introducing HPC to such researchers enables them to start using HPC. We gave talks to two groups of non-ICT researchers that introduced basic concepts focusing on the necessary practical steps needed to use HPC on a specific cluster. We also offered hands-on trainings for one of the groups which aimed to guide participants through the first steps of using HPC. Participants filled out questionnaires partly based on Kirkpatrick's training evaluation model before and after the talk, and after the hands-on training. We found that the talk increased participants' self-reported likelihood of using HPC in their future research, but this was not significant for the group where participation was voluntary. On the contrary, very few researchers participated in the hands-on training, and for these participants neither the talk, nor the hands-on training changed their self-reported likelihood of using HPC in their future research. We argue that our findings show that academia and researchers would benefit from an environment that not only expects researchers to train themselves, but provides structural support for acquiring new skills.

Peptides with Dual Antimicrobial-Anticancer Activity: Strategies to Overcome Peptide Limitations and Rational Design of Anticancer Peptides

Peptides are naturally produced by all organisms and exhibit a wide range of physiological, immunomodulatory, and wound healing functions. Furthermore, they can provide with protection against microorganisms and tumor cells. Their multifaceted performance, high selectivity, and reduced toxicity have positioned them as effective therapeutic agents, representing a positive economic impact for pharmaceutical companies. Currently, efforts have been made to invest in the development of new peptides with antimicrobial and anticancer properties, but the poor stability of these molecules in physiological environments has triggered a bottleneck. Therefore, some tools, such as nanotechnology and in silico approaches can be applied as alternatives to try to overcome these obstacles. In silico studies provide a priori knowledge that can lead to the development of new anticancer peptides with enhanced biological activity and improved stability. This review focuses on the current status of research in peptides with dual antimicrobial-anticancer activity, including advances in computational biology using in silico analyses as a powerful tool for the study and rational design of these types of peptides.