How to evaluate an agent's behavior to infrequent events?-Reliable performance estimation insensitive to class distribution

Vigilance to Painful Laser Stimuli is Associated with Increased State Anxiety and Tense Arousal

Introduction

Pain is frequently accompanied by enhanced arousal and hypervigilance to painful sensations. Here, we describe our findings in an experimental vigilance task requiring healthy participants to indicate when randomly timed moderately painful stimuli occur in a long train of mildly painful stimuli.

Methods

During a continuous performance task with painful laser stimuli (CPTpain), 18 participants rated pain intensity, unpleasantness, and salience. We tested for a vigilance decrement over time using classical metrics including correct targets (hits), incorrectly identified non-targets (false alarms), hit reaction time, and false alarm reaction time. We measured state anxiety and tense arousal before and after the task.

Results

We found a vigilance decrement across four 12.5-minute blocks of painful laser stimuli in hits [F3,51=2.91; p=0.043; time block 1>block 4 (t=2.77; p=0.035)]. Both self-report state anxiety (tpaired,17=3.34; p=0.0039) and tense arousal (tpaired,17=3.20; p=0.0053) increased after the task. We found a vigilance decrement during our laser pain vigilance task consistent with vigilance decrements found in other stimulus modalities. Furthermore, state anxiety positively correlated with tense arousal.

Discussion

CPTpain acutely increased tense arousal and state anxiety, consistent with previous results implicating the reciprocal interaction of state anxiety and acute painful sensations and the role of pain in augmenting tense arousal. These results may indicate a psychological process which predisposes the hypervigilant to developing greater acute pain, resulting in positive feedback, greater pain and anxiety.

Class imbalance should not throw you off balance: Choosing the right classifiers and performance metrics for brain decoding with imbalanced data

Machine learning (ML) is increasingly used in cognitive, computational and clinical neuroscience. The reliable and efficient application of ML requires a sound understanding of its subtleties and limitations. Training ML models on datasets with imbalanced classes is a particularly common problem, and it can have severe consequences if not adequately addressed. With the neuroscience ML user in mind, this paper provides a didactic assessment of the class imbalance problem and illustrates its impact through systematic manipulation of data imbalance ratios in (i) simulated data and (ii) brain data recorded with electroencephalography (EEG), magnetoencephalography (MEG) and functional magnetic resonance imaging (fMRI). Our results illustrate how the widely-used Accuracy (Acc) metric, which measures the overall proportion of successful predictions, yields misleadingly high performances, as class imbalance increases. Because Acc weights the per-class ratios of correct predictions proportionally to class size, it largely disregards the performance on the minority class. A binary classification model that learns to systematically vote for the majority class will yield an artificially high decoding accuracy that directly reflects the imbalance between the two classes, rather than any genuine generalizable ability to discriminate between them. We show that other evaluation metrics such as the Area Under the Curve (AUC) of the Receiver Operating Characteristic (ROC), and the less common Balanced Accuracy (BAcc) metric - defined as the arithmetic mean between sensitivity and specificity, provide more reliable performance evaluations for imbalanced data. Our findings also highlight the robustness of Random Forest (RF), and the benefits of using stratified cross-validation and hyperprameter optimization to tackle data imbalance. Critically, for neuroscience ML applications that seek to minimize overall classification error, we recommend the routine use of BAcc, which in the specific case of balanced data is equivalent to using standard Acc, and readily extends to multi-class settings. Importantly, we present a list of recommendations for dealing with imbalanced data, as well as open-source code to allow the neuroscience community to replicate and extend our observations and explore alternative approaches to coping with imbalanced data.

PToPI: A Comprehensive Review, Analysis, and Knowledge Representation of Binary Classification Performance Measures/Metrics

Although few performance evaluation instruments have been used conventionally in different machine learning-based classification problem domains, there are numerous ones defined in the literature. This study reviews and describes performance instruments via formally defined novel concepts and clarifies the terminology. The study first highlights the issues in performance evaluation via a survey of 78 mobile-malware classification studies and reviews terminology. Based on three research questions, it proposes novel concepts to identify characteristics, similarities, and differences of instruments that are categorized into 'performance measures' and 'performance metrics' in the classification context for the first time. The concepts reflecting the intrinsic properties of instruments such as canonical form, geometry, duality, complementation, dependency, and leveling, aim to reveal similarities and differences of numerous instruments, such as redundancy and ground-truth versus prediction focuses. As an application of knowledge representation, we introduced a new exploratory table called PToPI (Periodic Table of Performance Instruments) for 29 measures and 28 metrics (69 instruments including variant and parametric ones). Visualizing proposed concepts, PToPI provides a new relational structure for the instruments including graphical, probabilistic, and entropic ones to see their properties and dependencies all in one place. Applications of the exploratory table in six examples from different domains in the literature have shown that PToPI aids overall instrument analysis and selection of the proper performance metrics according to the specific requirements of a classification problem. We expect that the proposed concepts and PToPI will help researchers comprehend and use the instruments and follow a systematic approach to classification performance evaluation and publication.

Predictive Fraud Analysis Applying the Fraud Triangle Theory through Data Mining Techniques

Fraud is increasingly common, and so are the losses caused by this phenomenon. There is, thus, an essential economic incentive to study this problem, particularly fraud prevention. One barrier complicating the research in this direction is the lack of public data sets that embed fraudulent activities. In addition, although efforts have been made to detect fraud using machine learning, such actions have not considered the component of human behavior when detecting fraud. We propose a mechanism to detect potential fraud by analyzing human behavior within a data set in this work. This approach combines a predefined topic model and a supervised classifier to generate an alert from the possible fraud-related text. Potential fraud would be detected based on a model built from such a classifier. As a result of this work, a synthetic fraud-related data set is made. Four topics associated with the vertices of the fraud triangle theory are unveiled when assessing different topic modeling techniques. After benchmarking topic modeling techniques and supervised and deep learning classifiers, we find that LDA, random forest, and CNN have the best performance in this scenario. The results of our work suggest that our approach is feasible in practice since several such models obtain an average AUC higher than 0.8. Namely, the fraud triangle theory combined with topic modeling and linear classifiers could provide a promising framework for predictive fraud analysis.

Single-Option P300-BCI Performance Is Affected by Visual Stimulation Conditions

The P300 paradigm is one of the most promising techniques for its robustness and reliability in Brain-Computer Interface (BCI) applications, but it is not exempt from shortcomings. The present work studied single-trial classification effectiveness in distinguishing between target and non-target responses considering two conditions of visual stimulation and the variation of the number of symbols presented to the user in a single-option visual frame. In addition, we also investigated the relationship between the classification results of target and non-target events when training and testing the machine-learning model with datasets containing different stimulation conditions and different number of symbols. To this end, we designed a P300 experimental protocol considering, as conditions of stimulation: the color highlighting or the superimposing of a cartoon face and from four to nine options. These experiments were carried out with 19 healthy subjects in 3 sessions. The results showed that the Event-Related Potentials (ERP) responses and the classification accuracy are stronger with cartoon faces as stimulus type and similar irrespective of the amount of options. In addition, the classification performance is reduced when using datasets with different type of stimulus, but it is similar when using datasets with different the number of symbols. These results have a special connotation for the design of systems, in which it is intended to elicit higher levels of evoked potentials and, at the same time, optimize training time.

Optimization of convolutional neural network hyperparameters for automatic classification of adult mosquitoes

The economic and social impacts due to diseases transmitted by mosquitoes in the latest years have been significant. Currently, no specific treatment or commercial vaccine exists for the control and prevention of arboviruses, thereby making entomological characterization fundamental in combating diseases such as dengue, chikungunya, and Zika. The morphological identification of mosquitos includes a visual exam of the samples. It is time consuming and requires adequately trained professionals. Accordingly, the development of a new automated method for realizing mosquito-perception and -classification is becoming increasingly essential. Therefore, in this study, a computational model based on a convolutional neural network (CNN) was developed to extract features from the images of mosquitoes and then classify the species Aedes aegypti, Aedes albopictus, and Culex quinquefasciatus. In addition, the model was trained to detect the mosquitoes of the genus Aedes. To train CNNs to perform the automatic morphological classification of mosquitoes, a dataset, which included 7,561 images of the target mosquitoes and 1,187 images of other insects, was acquired. Various neural networks, such as Xception and DenseNet, were used for developing the automatic-classification model based on images. A structured optimization process of random search and grid search was developed to select the hyperparameters set and increase the accuracy of the model. In addition, strategies to eliminate overfitting were implemented to increase the generalization of the model. The optimized model, during the test phase, obtained the balanced accuracy (BA) of 93.5% in classifying the target mosquitoes and other insects and the BA of 97.3% in detecting the mosquitoes of the genus Aedes in comparison to Culex. The results provide fundamental information for performing the automatic morphological classification of mosquito species. Using a CNN-embedded entomological tool is a valuable and accessible resource for health workers and non-taxonomists for identifying insects that can transmit infectious diseases.

Gait Recognition via Deep Learning of the Center-of-Pressure Trajectory

The fact that every human has a distinctive walking style has prompted a proposal to use gait recognition as an identification criterion. Using end-to-end learning, I investigated whether the center-of-pressure (COP) trajectory is sufficiently unique to identify a person with high certainty. Thirty-six adults walked for 30 min on a treadmill equipped with a force platform that continuously recorded the positions of the COP. The raw two-dimensional signals were sliced into segments of two gait cycles. A set of 20,250 segments from 30 subjects was used to configure and train convolutional neural networks (CNNs). The best CNN classified a separate set containing 2250 segments with an overall accuracy of 99.9%. A second set of 4500 segments from the six remaining subjects was then used for transfer learning. Several small subsamples of this set were selected randomly and used to fine tune the pretrained CNNs. Training with two segments per subject was sufficient to achieve 100% accuracy. The results suggest that every person produces a unique trajectory of underfoot pressures while walking and that CNNs can learn the distinctive features of these trajectories. By applying a pretrained CNN (transfer learning), a couple of strides seem enough to learn and identify new gaits. However, these promising results should be confirmed in a larger sample under realistic conditions.

Empirical Comparison of Distributed Source Localization Methods for Single-Trial Detection of Movement Preparation

The development of technologies for the treatment of movement disorders, like stroke, is still of particular interest in brain-computer interface (BCI) research. In this context, source localization methods (SLMs), that reconstruct the cerebral origin of brain activity measured outside the head, e.g., via electroencephalography (EEG), can add a valuable insight into the current state and progress of the treatment. However, in BCIs SLMs were often solely considered as advanced signal processing methods that are compared against other methods based on the classification performance alone. Though, this approach does not guarantee physiological meaningful results. We present an empirical comparison of three established distributed SLMs with the aim to use one for single-trial movement prediction. The SLMs wMNE, sLORETA, and dSPM were applied on data acquired from eight subjects performing voluntary arm movements. Besides the classification performance as quality measure, a distance metric was used to asses the physiological plausibility of the methods. For the distance metric, which is usually measured to the source position of maximum activity, we further propose a variant based on clusters that is better suited for the single-trial case in which several sources are likely and the actual maximum is unknown. The two metrics showed different results. The classification performance revealed no significant differences across subjects, indicating that all three methods are equally well-suited for single-trial movement prediction. On the other hand, we obtained significant differences in the distance measure, favoring wMNE even after correcting the distance with the number of reconstructed clusters. Further, distance results were inconsistent with the traditional method using the maximum, indicating that for wMNE the point of maximum source activity often did not coincide with the nearest activation cluster. In summary, the presented comparison might help users to select an appropriate SLM and to understand the implications of the selection. The proposed methodology pays attention to the particular properties of distributed SLMs and can serve as a framework for further comparisons.

Multi-Tasking and Choice of Training Data Influencing Parietal ERP Expression and Single-Trial Detection-Relevance for Neuroscience and Clinical Applications

Event-related potentials (ERPs) are often used in brain-computer interfaces (BCIs) for communication or system control for enhancing or regaining control for motor-disabled persons. Especially results from single-trial EEG classification approaches for BCIs support correlations between single-trial ERP detection performance and ERP expression. Hence, BCIs can be considered as a paradigm shift contributing to new methods with strong influence on both neuroscience and clinical applications. Here, we investigate the relevance of the choice of training data and classifier transfer for the interpretability of results from single-trial ERP detection. In our experiments, subjects performed a visual-motor oddball task with motor-task relevant infrequent (targets), motor-task irrelevant infrequent (deviants), and motor-task irrelevant frequent (standards) stimuli. Under dual-task condition, a secondary senso-motor task was performed, compared to the simple-task condition. For evaluation, average ERP analysis and single-trial detection analysis with different numbers of electrodes were performed. Further, classifier transfer was investigated between simple and dual task. Parietal positive ERPs evoked by target stimuli (but not by deviants) were expressed stronger under dual-task condition, which is discussed as an increase of task emphasis and brain processes involved in task coordination and change of task set. Highest classification performance was found for targets irrespective whether all 62, 6 or 2 parietal electrodes were used. Further, higher detection performance of targets compared to standards was achieved under dual-task compared to simple-task condition in case of training on data from 2 parietal electrodes corresponding to results of ERP average analysis. Classifier transfer between tasks improves classification performance in case that training took place on more varying examples (from dual task). In summary, we showed that P300 and overlaying parietal positive ERPs can successfully be detected while subjects are performing additional ongoing motor activity. This supports single-trial detection of ERPs evoked by target events to, e.g., infer a patient's attentional state during therapeutic intervention.

CANcer-specific Evaluation System (CANES): a high-accuracy platform, for preclinical single/multi-biomarker discovery

The recent creation of enormous, cancer-related “Big Data” public depositories represents a powerful means for understanding tumorigenesis. However, a consistently accurate system for clinically evaluating single/multi-biomarkers remains lacking, and it has been asserted that oft-failed clinical advancement of biomarkers occurs within the very early stages of biomarker assessment. To address these challenges, we developed a clinically testable, web-based tool, CANcer-specific single/multi-biomarker Evaluation System (CANES), to evaluate biomarker effectiveness, across 2,134 whole transcriptome datasets, from 94,147 biological samples (from 18 tumor types). For user-provided single/multi-biomarkers, CANES evaluates the performance of single/multi-biomarker candidates, based on four classification methods, support vector machine, random forest, neural networks, and classification and regression trees. In addition, CANES offers several advantages over earlier analysis tools, including: 1) survival analysis; 2) evaluation of mature miRNAs as markers for user-defined diagnostic or prognostic purposes; and 3) provision of a “pan-cancer” summary view, based on each single marker. We believe that such “landscape” evaluation of single/multi-biomarkers, for diagnostic therapeutic/prognostic decision-making, will be highly valuable for the discovery and “repurposing” of existing biomarkers (and their specific targeted therapies), leading to improved patient therapeutic stratification, a key component of targeted therapy success for the avoidance of therapy resistance.

A Hybrid FPGA-Based System for EEG- and EMG-Based Online Movement Prediction

A current trend in the development of assistive devices for rehabilitation, for example exoskeletons or active orthoses, is to utilize physiological data to enhance their functionality and usability, for example by predicting the patient's upcoming movements using electroencephalography (EEG) or electromyography (EMG). However, these modalities have different temporal properties and classification accuracies, which results in specific advantages and disadvantages. To use physiological data analysis in rehabilitation devices, the processing should be performed in real-time, guarantee close to natural movement onset support, provide high mobility, and should be performed by miniaturized systems that can be embedded into the rehabilitation device. We present a novel Field Programmable Gate Array (FPGA) -based system for real-time movement prediction using physiological data. Its parallel processing capabilities allows the combination of movement predictions based on EEG and EMG and additionally a P300 detection, which is likely evoked by instructions of the therapist. The system is evaluated in an offline and an online study with twelve healthy subjects in total. We show that it provides a high computational performance and significantly lower power consumption in comparison to a standard PC. Furthermore, despite the usage of fixed-point computations, the proposed system achieves a classification accuracy similar to systems with double precision floating-point precision.

Rotational data augmentation for electroencephalographic data

MOTIVATION

For deep learning on image data, a common approach is to augment the training data by artificial new images, using techniques like moving windows, scaling, affine distortions, and elastic deformations. In contrast to image data, electroencephalographic (EEG) data suffers even more from the lack of sufficient training data.

METHODS

We suggest and evaluate rotational distortions similar to affine/rotational distortions of images to generate augmented data.

RESULTS

Our approach increases the performance of signal processing chains for EEG-based brain-computer interfaces when rotating only around y- and z-axis with an angle around ±18 degrees to generate new data.

CONCLUSION

This shows that our processing efficient approach generates meaningful data and encourages to look for further new methods for EEG data augmentation.

An Intelligent Man-Machine Interface-Multi-Robot Control Adapted for Task Engagement Based on Single-Trial Detectability of P300

Advanced man-machine interfaces (MMIs) are being developed for teleoperating robots at remote and hardly accessible places. Such MMIs make use of a virtual environment and can therefore make the operator immerse him-/herself into the environment of the robot. In this paper, we present our developed MMI for multi-robot control. Our MMI can adapt to changes in task load and task engagement online. Applying our approach of embedded Brain Reading we improve user support and efficiency of interaction. The level of task engagement was inferred from the single-trial detectability of P300-related brain activity that was naturally evoked during interaction. With our approach no secondary task is needed to measure task load. It is based on research results on the single-stimulus paradigm, distribution of brain resources and its effect on the P300 event-related component. It further considers effects of the modulation caused by a delayed reaction time on the P300 component evoked by complex responses to task-relevant messages. We prove our concept using single-trial based machine learning analysis, analysis of averaged event-related potentials and behavioral analysis. As main results we show (1) a significant improvement of runtime needed to perform the interaction tasks compared to a setting in which all subjects could easily perform the tasks. We show that (2) the single-trial detectability of the event-related potential P300 can be used to measure the changes in task load and task engagement during complex interaction while also being sensitive to the level of experience of the operator and (3) can be used to adapt the MMI individually to the different needs of users without increasing total workload. Our online adaptation of the proposed MMI is based on a continuous supervision of the operator's cognitive resources by means of embedded Brain Reading. Operators with different qualifications or capabilities receive only as many tasks as they can perform to avoid mental overload as well as mental underload.

An Adaptive Spatial Filter for User-Independent Single Trial Detection of Event-Related Potentials

GOAL

Current brain-computer interfaces (BCIs) are usually based on various, often supervised, signal processing methods. The disadvantage of supervised methods is the requirement to calibrate them with recently acquired subject-specific training data. Here, we present a novel algorithm for dimensionality reduction (spatial filter), that is ideally suited for single-trial detection of event-related potentials (ERPs) and can be adapted online to a new subject to minimize or avoid calibration time.

METHODS

The algorithm is based on the well-known xDAWN filter, but uses generalized eigendecomposition to allow an incremental training by recursive least squares (RLS) updates of the filter coefficients. We analyze the effectiveness of the spatial filter in different transfer scenarios and combinations with adaptive classifiers.

RESULTS

The results show that it can compensate changes due to switching between different users, and therefore allows to reuse training data that has been previously recorded from other subjects.

CONCLUSIONS

The presented approach allows to reduce or completely avoid a calibration phase and to instantly use the BCI system with only a minor decrease of performance.

SIGNIFICANCE

The novel filter can adapt a precomputed spatial filter to a new subject and make a BCI system user independent.