Outlier Detection Using Replicator Neural Networks

Anomaly detection via Gumbel Noise Score Matching

We propose Gumbel Noise Score Matching (GNSM), a novel unsupervised method to detect anomalies in categorical data. GNSM accomplishes this by estimating the scores, i.e., the gradients of log likelihoods w.r.t. inputs, of continuously relaxed categorical distributions. We test our method on a suite of anomaly detection tabular datasets. GNSM achieves a consistently high performance across all experiments. We further demonstrate the flexibility of GNSM by applying it to image data where the model is tasked to detect poor segmentation predictions. Images ranked anomalous by GNSM show clear segmentation failures, with the anomaly scores strongly correlating with segmentation metrics computed on ground-truth. We outline the score matching training objective utilized by GNSM and provide an open-source implementation of our work.

MFAM-AD: an anomaly detection model for multivariate time series using attention mechanism to fuse multi-scale features

Multivariate time series anomaly detection has garnered significant attention in fields such as IT operations, finance, medicine, and industry. However, a key challenge lies in the fact that anomaly patterns often exhibit multi-scale temporal variations, which existing detection models often fail to capture effectively. This limitation significantly impacts detection accuracy. To address this issue, we propose the MFAM-AD model, which combines the strengths of convolutional neural networks (CNNs) and bi-directional long short-term memory (Bi-LSTM). The MFAM-AD model is designed to enhance anomaly detection accuracy by seamlessly integrating temporal dependencies and multi-scale spatial features. Specifically, it utilizes parallel convolutional layers to extract features across different scales, employing an attention mechanism for optimal feature fusion. Additionally, Bi-LSTM is leveraged to capture time-dependent information, reconstruct the time series and enable accurate anomaly detection based on reconstruction errors. In contrast to existing algorithms that struggle with inadequate feature fusion or are confined to single-scale feature analysis, MFAM-AD effectively addresses the unique challenges of multivariate time series anomaly detection. Experimental results on five publicly available datasets demonstrate the superiority of the proposed model. Specifically, on the datasets SMAP, MSL, and SMD1-1, our MFAM-AD model has the second-highest F1 score after the current state-of-the-art DCdetector model. On the datasets NIPS-TS-SWAN and NIPS-TS-GECCO, the F1 scores of MAFM-AD are 0.046 (6.2%) and 0.09 (21.3%) higher than those of DCdetector, respectively(the value ranges from 0 to 1). These findings validate the MFAMAD model's efficacy in multivariate time series anomaly detection, highlighting its potential in various real-world applications.

Outlier analysis for accelerating clinical discovery: An augmented intelligence framework and a systematic review

Clinical discoveries largely depend on dedicated clinicians and scientists to identify and pursue unique and unusual clinical encounters with patients and communicate these through case reports and case series. This process has remained essentially unchanged throughout the history of modern medicine. However, these traditional methods are inefficient, especially considering the modern-day availability of health-related data and the sophistication of computer processing. Outlier analysis has been used in various fields to uncover unique observations, including fraud detection in finance and quality control in manufacturing. We propose that clinical discovery can be formulated as an outlier problem within an augmented intelligence framework to be implemented on any health-related data. Such an augmented intelligence approach would accelerate the identification and pursuit of clinical discoveries, advancing our medical knowledge and uncovering new therapies and management approaches. We define clinical discoveries as contextual outliers measured through an information-based approach and with a novelty-based root cause. Our augmented intelligence framework has five steps: define a patient population with a desired clinical outcome, build a predictive model, identify outliers through appropriate measures, investigate outliers through domain content experts, and generate scientific hypotheses. Recognizing that the field of obstetrics can particularly benefit from this approach, as it is traditionally neglected in commercial research, we conducted a systematic review to explore how outlier analysis is implemented in obstetric research. We identified two obstetrics-related studies that assessed outliers at an aggregate level for purposes outside of clinical discovery. Our findings indicate that using outlier analysis in clinical research in obstetrics and clinical research, in general, requires further development.

Hybrid Anomaly Detection in Time Series by Combining Kalman Filters and Machine Learning Models

Due to connectivity and automation trends, the medical device industry is experiencing increased demand for safety and security mechanisms. Anomaly detection has proven to be a valuable approach for ensuring safety and security in other industries, such as automotive or IT. Medical devices must operate across a wide range of values due to variations in patient anthropometric data, making anomaly detection based on a simple threshold for signal deviations impractical. For example, surgical robots directly contacting the patient's tissue require precise sensor data. However, since the deformation of the patient's body during interaction or movement is highly dependent on body mass, it is impossible to define a single threshold for implausible sensor data that applies to all patients. This also involves statistical methods, such as Z-score, that consider standard deviation. Even pure machine learning algorithms cannot be expected to provide the required accuracy simply due to the lack of available training data. This paper proposes using hybrid filters by combining dynamic system models based on expert knowledge and data-based models for anomaly detection in an operating room scenario. This approach can improve detection performance and explainability while reducing the computing resources needed on embedded devices, enabling a distributed approach to anomaly detection.

Scalable Learning Framework for Detecting New Types of Twitter Spam with Misuse and Anomaly Detection

The growing popularity of social media has engendered the social problem of spam proliferation through this medium. New spam types that evade existing spam detection systems are being developed continually, necessitating corresponding countermeasures. This study proposes an anomaly detection-based framework to detect new Twitter spam, which works by modeling the characteristics of non-spam tweets and using anomaly detection to classify tweets deviating from this model as anomalies. However, because modeling varied non-spam tweets is challenging, the technique's spam detection and false positive (FP) rates are low and high, respectively. To overcome this shortcoming, anomaly detection is performed on known spam tweets pre-detected using a trained decision tree while modeling normal tweets. A one-class support vector machine and an autoencoder with high detection rates are used for anomaly detection. The proposed framework exhibits superior detection rates for unknown spam compared to conventional techniques, while maintaining equivalent or improved detection and FP rates for known spam. Furthermore, the framework can be adapted to changes in spam conditions by adjusting the costs of detection errors.

Nested Binary Classifier as an Outlier Detection Method in Human Activity Recognition Systems

The present article is devoted to outlier detection in phases of human movement. The aim was to find the most efficient machine learning method to detect abnormal segments inside physical activities in which there is a probability of origin from other activities. The problem was reduced to a classification task. The new method is proposed based on a nested binary classifier. Test experiments were then conducted using several of the most popular machine learning algorithms (linear regression, support vector machine, k-nearest neighbor, decision trees). Each method was separately tested on three datasets varying in characteristics and number of records. We set out to evaluate the effectiveness of the models, basic measures of classifier evaluation, and confusion matrices. The nested binary classifier was compared with deep neural networks. Our research shows that the method of nested binary classifiers can be considered an effective way of recognizing outlier patterns for HAR systems.

A Spatiotemporal Deep Learning Approach for Unsupervised Anomaly Detection in Cloud Systems

Anomaly detection is a critical task for maintaining the performance of a cloud system. Using data-driven methods to address this issue is the mainstream in recent years. However, due to the lack of labeled data for training in practice, it is necessary to enable an anomaly detection model trained on contaminated data in an unsupervised way. Besides, with the increasing complexity of cloud systems, effectively organizing data collected from a wide range of components of a system and modeling spatiotemporal dependence among them become a challenge. In this article, we propose TopoMAD, a stochastic seq2seq model which can robustly model spatial and temporal dependence among contaminated data. We include system topological information to organize metrics from different components and apply sliding windows over metrics collected continuously to capture the temporal dependence. We extract spatial features with the help of graph neural networks and temporal features with long short-term memory networks. Moreover, we develop our model based on variational auto-encoder, enabling it to work well robustly even when trained on contaminated data. Our approach is validated on the run-time performance data collected from two representative cloud systems, namely, a big data batch processing system and a microservice-based transaction processing system. The experimental results show that TopoMAD outperforms some state-of-the-art methods on these two data sets.

Anomaly detection in radiotherapy plans using deep autoencoder networks

Purpose

Treatment plans are used for patients under radiotherapy in clinics. Before execution, these plans are checked for safety and quality by human experts. A few of them were identified with flaws and needed further improvement. To automate this checking process, an unsupervised learning method based on an autoencoder was proposed.

Methods

First, features were extracted from the treatment plan by human experts. Then, these features were assembled and used for model learning. After network optimization, a reconstruction error between the predicted and target signals was obtained. Finally, the questionable plans were identified based on the value of the reconstruction error. A large value of the reconstruction error indicates a longer distance from the standard distribution of normal plans. A total of 576 treatment plans for breast cancer patients were used for the test. Among them, 19 were questionable plans identified by human experts. To evaluate the performance of the autoencoder, it was compared with four baseline detection algorithms, namely, local outlier factor (LOF), hierarchical density-based spatial clustering of applications with noise (HDBSCAN), one-class support vector machine (OC-SVM), and principal component analysis (PCA).

Results

The results showed that the autoencoder achieved the best performance than the other four baseline algorithms. The AUC value of the autoencoder was 0.9985, while the second one was 0.9535 (LOF). While maintaining 100% recall, the average accuracy and precision of the results by the autoencoder were 0.9658 and 0.5143, respectively. While maintaining 100% recall, the average accuracy and precision of the results by LOF were 0.8090 and 0.1472, respectively.

Conclusion

The autoencoder can effectively identify questionable plans from a large group of normal plans. There is no need to label the data and prepare the training data for model learning. The autoencoder provides an effective way to carry out an automatic plan checking in radiotherapy.

Fluctuation-based outlier detection

Outlier detection is an important topic in machine learning and has been used in a wide range of applications. Outliers are objects that are few in number and deviate from the majority of objects. As a result of these two properties, we show that outliers are susceptible to a mechanism called fluctuation. This article proposes a method called fluctuation-based outlier detection (FBOD) that achieves a low linear time complexity and detects outliers purely based on the concept of fluctuation without employing any distance, density or isolation measure. Fundamentally different from all existing methods. FBOD first converts the Euclidean structure datasets into graphs by using random links, then propagates the feature value according to the connection of the graph. Finally, by comparing the difference between the fluctuation of an object and its neighbors, FBOD determines the object with a larger difference as an outlier. The results of experiments comparing FBOD with eight state-of-the-art algorithms on eight real-worlds tabular datasets and three video datasets show that FBOD outperforms its competitors in the majority of cases and that FBOD has only 5% of the execution time of the fastest algorithm. The experiment codes are available at: https://github.com/FluctuationOD/Fluctuation-based-Outlier-Detection .

Anomaly Detection of DC Nut Runner Processes in Engine Assembly

In many manufacturing systems, anomaly detection is critical to identifying process errors and ensuring product quality. This paper proposes three semi-supervised solutions to detect anomalies in Direct Current (DC) Nut Runner engine assembly processes. The nut runner process is a challenging anomaly detection problem due to the manual nature of the process inducing high variability and ambiguity of the anomalous class. These characteristics lead to a scenario where anomalies are not outliers, and the normal operating conditions are difficult to define. To address these challenges, a Gaussian Mixture Model (GMM) was trained using a semi-supervised approach. Three dimensionality reduction methods were compared in pre-processing: PCA, t-SNE, and UMAP. These approaches are demonstrated to outperform the current approaches used by a major automotive company on two real-world datasets. Furthermore, a novel approach to labelling real-world data is proposed, including the concept of an ‘Anomaly No Concern’ class, in addition to the traditional labels of ‘Anomaly’ and ‘Normal’. Introducing this new term helped address knowledge gaps between data scientists and domain experts, as well as providing new insights during model development and testing. This represents a major advancement in identifying anomalies in manual production processes that use handheld tools.

A Self-Calibrating Localization Solution for Sport Applications with UWB Technology

This study addressed the problem of localization in an ultrawide-band (UWB) network, where the positions of both the access points and the tags needed to be estimated. We considered a fully wireless UWB localization system, comprising both software and hardware, featuring easy plug-and-play usability for the consumer, primarily targeting sport and leisure applications. Anchor self-localization was addressed by two-way ranging, also embedding a Gauss-Newton algorithm for the estimation and compensation of antenna delays, and a modified isolation forest algorithm working with low-dimensional set of measurements for outlier identification and removal. This approach avoids time-consuming calibration procedures, and it enables accurate tag localization by the multilateration of time difference of arrival measurements. For the assessment of performance and the comparison of different algorithms, we considered an experimental campaign with data gathered by a proprietary UWB localization system.

SEHIDS: Self Evolving Host-Based Intrusion Detection System for IoT Networks

The Internet of Things (IoT) offers unprecedented opportunities to access anything from anywhere and at any time. It is, therefore, not surprising that the IoT acts as a paramount infrastructure for most modern and envisaged systems, including but not limited to smart homes, e-health, and intelligent transportation systems. However, the prevalence of IoT networks and the important role they play in various critical aspects of our lives make them a target for various types of advanced cyberattacks: Dyn attack, BrickerBot, Sonic, Smart Deadbolts, and Silex are just a few examples. Motivated by the need to protect IoT networks, this paper proposes SEHIDS: Self Evolving Host-based Intrusion Detection System. The underlying approach of SEHIDS is to equip each IoT node with a simple Artificial Neural Networks (ANN) architecture and a lightweight mechanism through which an IoT device can train this architecture online and evolves it whenever its performance prediction is degraded. By this means, SEHIDS enables each node to generate the ANN architecture required to detect the threats it faces, which makes SEHIDS suitable for the heterogeneity and turbulence of traffic amongst nodes. Moreover, the gradual evolution of the SEHIDS architecture facilitates retaining it to its near-minimal configurations, which saves the resources required to compute, store, and manipulate the model's parameters and speeds up the convergence of the model to the zero-classification regions. It is noteworthy that SEHIDS specifies the evolving criteria based on the outcomes of the built-in model's loss function, which is, in turn, facilitates using SEHIDS to develop the two common types of IDS: signature-based and anomaly-based. Where in the signature-based IDS version, a supervised architecture (i.e., multilayer perceptron architecture) is used to classify different types of attacks, while in the anomaly-based IDS version, an unsupervised architecture (i.e., replicator neuronal network) is used to distinguish benign from malicious traffic. Comprehensive assessments for SEHIDS from different perspectives were conducted with three recent datasets containing a variety of cyberattacks targeting IoT networks: BoT-IoT, TON-IOT, and IoTID20. These results of assessments demonstrate that SEHIDS is able to make accurate predictions of 1 True Positive and is suitable for IoT networks with the order of small fractions of the resources of typical IoT devices.

Bounding open space risk with decoupling autoencoders in open set recognition

One-vs-Rest (OVR) classification aims to distinguish a single class of interest (COI) from other classes. The concept of novelty detection and robustness to dataset shift becomes crucial in OVR when the scope of the rest class is extended from the classes observed during training to unseen and possibly unrelated classes, a setting referred to as open set recognition (OSR). In this work, we propose a novel architecture, namely decoupling autoencoder (DAE), which provides a proven upper bound on the open space risk and minimizes open space risk via a dedicated training routine. Our method is benchmarked within three different scenarios, each isolating different aspects of OSR, namely plain classification, outlier detection, and dataset shift. The results conclusively show that DAE achieves robust performance across all three tasks. This level of cross-task robustness is not observed for any of the seven potent baselines from the OSR, OVR, outlier detection, and ensembling domain which, apart from ATA (Lübbering et al., From imbalanced classification to supervised outlier detection problems: adversarially trained auto encoders. In: Artificial neural networks and machine learning—ICANN 2020, 2020), tend to fail on either one of the tasks. Similar to DAE, ATA is based on autoencoders and facilitates the reconstruction error to predict the inlierness of a sample. However unlike DAE, it does not provide any uncertainty scores and therefore lacks rudimentary means of interpretation. Our adversarial robustness and local stability results further support DAE’s superiority in the OSR setting, emphasizing its applicability in safety-critical systems.

Memory-Augmented Generative Adversarial Networks for Anomaly Detection

We propose a memory-augmented deep learning model for semisupervised anomaly detection (AD). While many traditional AD methods focus on modeling the distribution of normal data, additional constraints in the modeling process are needed to distinguish between normal and abnormal data. The proposed model, named memory augmented generative adversarial networks (MEMGAN), is coupled with external memory units through attentional operations. One property of MEMGAN in the latent space is such that encoded normal data are expected to reside in the convex hull of the memory units, while the abnormal ones are separated outside. This property makes the AD process of MEMGAN more robust and reliable. Experiments on AD datasets adapted from MVTec, MNIST, CIFAR10, and Arrhythmia demonstrate that MEMGAN notably improves over previous AD models. We also find that the decoded memory units in MEMGAN are more diverse and interpretable than those in previous memory-augmented models.

$${{\mathrm {Latent}}Out}$$: an unsupervised deep anomaly detection approach exploiting latent space distribution

Anomaly detection methods exploiting autoencoders (AE) have shown good performances. Unfortunately, deep non-linear architectures are able to perform high dimensionality reduction while keeping reconstruction error low, thus worsening outlier detecting performances of AEs. To alleviate the above problem, recently some authors have proposed to exploit Variational autoencoders (VAE) and bidirectional Generative Adversarial Networks (GAN), which arise as a variant of standard AEs designed for generative purposes, both enforcing the organization of the latent space guaranteeing continuity. However, these architectures share with standard AEs the problem that they generalize so well that they can also well reconstruct anomalies. In this work we argue that the approach of selecting the worst reconstructed examples as anomalies is too simplistic if a continuous latent space autoencoder-based architecture is employed. We show that outliers tend to lie in the sparsest regions of the combined latent/error space and propose the $$\mathrm{VAE}Out$$ VAE O u t and $${{\mathrm {Latent}}Out}$$ Latent O u t unsupervised anomaly detection algorithms, identifying outliers by performing density estimation in this augmented feature space. The proposed approach shows sensible improvements in terms of detection performances over the standard approach based on the reconstruction error.

Convolutional Prototype Network for Open Set Recognition

Despite the success of convolutional neural network (CNN) in conventional closed-set recognition (CSR), it still lacks robustness for dealing with unknowns (those out of known classes) in open environment. To improve the robustness of CNN in open-set recognition (OSR) and meanwhile maintain its high accuracy in CSR, we propose an alternative deep framework called convolutional prototype network (CPN), which keeps CNN for representation learning but replaces the closed-world assumed softmax with an open-world oriented and human-like prototype model. To equip CPN with discriminative ability for classifying known samples, we design several discriminative losses for training. Moreover, to increase the robustness of CPN for unknowns, we interpret CPN from the perspective of generative model and further propose a generative loss, which is essentially maximizing the log-likelihood of known samples and serves as a latent regularization for discriminative learning. The combination of discriminative and generative losses makes CPN a hybrid model with advantages for both CSR and OSR. Under the designed losses, the CPN is trained end-to-end for learning the convolutional network and prototypes jointly. For application of CPN in OSR, we propose two rejection rules for detecting different types of unknowns. Experiments on several datasets demonstrate the efficiency and effectiveness of CPN for both CSR and OSR tasks.

Outlier detection in multimodal MRI identifies rare individual phenotypes among more than 15,000 brains

Outliers in neuroimaging represent spurious data or the data of unusual phenotypes that deserve special attention such as clinical follow-up. Outliers have usually been detected in a supervised or semi-supervised manner for labeled neuroimaging cohorts. There has been much less work using unsupervised outlier detection on large unlabeled cohorts like the UK Biobank brain imaging dataset. Given its large sample size, rare imaging phenotypes within this unique cohort are of interest, as they are often clinically relevant and could be informative for discovering new processes. Here, we developed a two-level outlier detection and screening methodology to characterize individual outliers from the multimodal MRI dataset of more than 15,000 UK Biobank subjects. In primary screening, using brain ventricles, white matter, cortical thickness, and functional connectivity-based imaging phenotypes, every subject was parameterized with an outlier score per imaging phenotype. Outlier scores of these imaging phenotypes had good-to-excellent test-retest reliability, with the exception of resting-state functional connectivity (RSFC). Due to the low reliability of RSFC outlier scores, RSFC outliers were excluded from further individual-level outlier screening. In secondary screening, the extreme outliers (1,026 subjects) were examined individually, and those arising from data collection/processing errors were eliminated. A representative subgroup of 120 subjects from the remaining non-artifactual outliers were radiologically reviewed, and radiological findings were identified in 97.5% of them. This study establishes an unsupervised framework for investigating rare individual imaging phenotypes within a large neuroimaging cohort.

Outlier detection for incomplete real-valued data based on inner boundary

Outlier detection is a process to find out the objects that have the abnormal behavior. It can be applied in many aspects, such as public security, finance and medical care. An information system (IS) as a database that shows relationships between objects and attributes. A real-valued information system (RVIS) is an IS whose information values are real numbers. A RVIS with missing values is an incomplete real-valued information system (IRVIS). The notion of inner boundary comes from the boundary region in rough set theory (RST). This paper conducts experiments directly in an IRVIS and investigates outlier detection in an IRVIS based on inner boundary. Firstly, the distance between two information values on each attribute of an IRVIS is introduced, and the parameter λ to control the distance is given. Then, the tolerance relations on the object set are defined according to the distance, by the way, the tolerance classes, the λ-lower and λ-upper approximations in an IRVIS are put forward. Next, the inner boundary under each conditional attribute in an IRVIS is presented. The more inner boundaries an object belongs to, the more likely it is to be an outlier. Finally, an outlier detection method in an IRVIS based on inner boundary is proposed, and the corresponding algorithm (DE) is designed, where DE means degree of exceptionality. Through the experiments base on UCI Machine Learning Repository data sets, the DE algorithm is compared with other five algorithms. Experimental results show that DE algorithm has the better outlier detection effect in an IRVIS. It is worth mentioning that for comprehensive comparison, ROC curve and AUC value are used to illustrate the advantages of the DE algorithm.

An end-to-end computer vision methodology for quantitative metallography

Metallography is crucial for a proper assessment of material properties. It mainly involves investigating the spatial distribution of grains and the occurrence and characteristics of inclusions or precipitates. This work presents a holistic few-shot artificial intelligence model for Quantitative Metallography, including Anomaly Detection, that automatically quantifies the degree of the anomaly of impurities in alloys. We suggest the following examination process: (1) deep semantic segmentation is performed on the inclusions (based on a suitable metallographic dataset of alloys and corresponding tags of inclusions), producing inclusions masks that are saved into a separated dataset. (2) Deep image inpainting is performed to fill the removed inclusions parts, resulting in 'clean' metallographic images, which contain the background of grains. (3) Grains' boundaries are marked using deep semantic segmentation (based on another metallographic dataset of alloys), producing boundaries that are ready for further inspection on the distribution of grains' size. (4) Deep anomaly detection and pattern recognition is performed on the inclusions masks to determine spatial, shape, and area anomaly detection of the inclusions. Finally, the end-to-end model recommends an expert on areas of interest for further examination. The physical result can re-tune the model according to the specific material at hand. Although the techniques presented here were developed for metallography analysis, most of them can be generalized to a broader set of microscopy problems that require automation. All source-codes as well as the datasets that were created for this work, are publicly available at https://github.com/Scientific-Computing-Lab-NRCN/MLography .

An iterative approach to unsupervised outlier detection using ensemble method and distance-based data filtering

Outlier or anomaly detection is the process through which datum/data with different properties from the rest of the data is/are identified. Their importance lies in their use in various domains such as fraud detection, network intrusion detection, and spam filtering. In this paper, we introduce a new outlier detection algorithm based on an ensemble method and distance-based data filtering with an iterative approach to detect outliers in unlabeled data. The ensemble method is used to cluster the unlabeled data and to filter out potential isolated outliers from the same by iteratively using a cluster membership threshold until the Dunn index score for clustering is maximized. The distance-based data filtering, on the other hand, removes the potential outlier clusters from the post-clustered data based on a distance threshold using the Euclidean distance measure of each data point from the majority cluster as the filtering factor. The performance of our algorithm is evaluated by applying it to 10 real-world machine learning datasets. Finally, we compare the results of our algorithm to various supervised and unsupervised outlier detection algorithms using Precision@n and F-score evaluation metrics.

Anomaly Pattern Detection in Streaming Data Based on the Transformation to Multiple Binary-Valued Data Streams

Anomaly pattern detection in a data stream aims to detect a time point where outliers begin to occur abnormally. Recently, a method for anomaly pattern detection has been proposed based on binary classification for outliers and statistical tests in the data stream of binary labels of normal or an outlier. It showed that an anomaly pattern can be detected accurately even when outlier detection performance is relatively low. However, since the anomaly pattern detection method is based on the binary classification for outliers, most well-known outlier detection methods, with the output of real-valued outlier scores, can not be used directly. In this paper, we propose an anomaly pattern detection method in a data stream using the transformation to multiple binary-valued data streams from real-valued outlier scores. By using three outlier detection methods, Isolation Forest(IF), Autoencoder-based outlier detection, and Local outlier factor(LOF), the proposed anomaly pattern detection method is tested using artificial and real data sets. The experimental results show that anomaly pattern detection using Isolation Forest gives the best performance.

Fast-RCM: Fast Tree-Based Unsupervised Rare-Class Mining

Rare classes are usually hidden in an imbalanced dataset with the majority of the data examples from major classes. Rare-class mining (RCM) aims at extracting all the data examples belonging to rare classes. Most of the existing approaches for RCM require a certain amount of labeled data examples as input. However, they are ineffective in practice since requesting label information from domain experts is time consuming and human-labor extensive. Thus, we investigate the unsupervised RCM problem, which to the best of our knowledge is the first such attempt. To this end, we propose an efficient algorithm called Fast-RCM for unsupervised RCM, which has an approximately linear time complexity with respect to data size and data dimensionality. Given an unlabeled dataset, Fast-RCM mines out the rare class by first building a rare tree for the input dataset and then extracting data examples of the rare classes based on this rare tree. Compared with the existing approaches which have quadric or even cubic time complexity, Fast-RCM is much faster and can be extended to large-scale datasets. The experimental evaluation on both synthetic and real-world datasets demonstrate that our algorithm can effectively and efficiently extract the rare classes from an unlabeled dataset under the unsupervised settings, and is approximately five times faster than that of the state-of-the-art methods.

On the nature and types of anomalies: a review of deviations in data

Anomalies are occurrences in a dataset that are in some way unusual and do not fit the general patterns. The concept of the anomaly is typically ill defined and perceived as vague and domain-dependent. Moreover, despite some 250 years of publications on the topic, no comprehensive and concrete overviews of the different types of anomalies have hitherto been published. By means of an extensive literature review this study therefore offers the first theoretically principled and domain-independent typology of data anomalies and presents a full overview of anomaly types and subtypes. To concretely define the concept of the anomaly and its different manifestations, the typology employs five dimensions: data type, cardinality of relationship, anomaly level, data structure, and data distribution. These fundamental and data-centric dimensions naturally yield 3 broad groups, 9 basic types, and 63 subtypes of anomalies. The typology facilitates the evaluation of the functional capabilities of anomaly detection algorithms, contributes to explainable data science, and provides insights into relevant topics such as local versus global anomalies.

Qualitative Data Clustering to Detect Outliers

Detecting outliers is a widely studied problem in many disciplines, including statistics, data mining, and machine learning. All anomaly detection activities are aimed at identifying cases of unusual behavior compared to most observations. There are many methods to deal with this issue, which are applicable depending on the size of the data set, the way it is stored, and the type of attributes and their values. Most of them focus on traditional datasets with a large number of quantitative attributes. The multitude of solutions related to detecting outliers in quantitative sets, a large and still has a small number of research solutions is a problem detecting outliers in data containing only qualitative variables. This article was designed to compare three different categorical data clustering algorithms: K-modes algorithm taken from MacQueen’s K-means algorithm and the STIRR and ROCK algorithms. The comparison concerned the method of dividing the set into clusters and, in particular, the outliers detected by algorithms. During the research, the authors analyzed the clusters detected by the indicated algorithms, using several datasets that differ in terms of the number of objects and variables. They have conducted experiments on the parameters of the algorithms. The presented study made it possible to check whether the algorithms similarly detect outliers in the data and how much they depend on individual parameters and parameters of the set, such as the number of variables, tuples, and categories of a qualitative variable.

The accuracy versus interpretability trade-off in fraud detection model

Like a hydra, fraudsters adapt and circumvent increasingly sophisticated barriers erected by public or private institutions. Among these institutions, banks must quickly take measures to avoid losses while guaranteeing the satisfaction of law-abiding customers. Facing an expanding flow of operations, effective banking relies on data analytics to support established risk control processes, but also on a better understanding of the underlying fraud mechanism. In addition, fraud being a criminal offence, the evidential aspect of the process must also be considered. These legal, operational, and strategic constraints lead to compromises on the means to be implemented for fraud management. This paper first focuses on the translation of practical questions raised in the banking industry at each step of the fraud management process into performance evaluation required to design a fraud detection model. Secondly, it considers a range of machine learning approaches that address these specificities: the imbalance between fraudulent and nonfraudulent operations, the lack of fully trusted labels, the concept-drift phenomenon, and the unavoidable trade-off between accuracy and interpretability of detection. This state-of-the-art review sheds some light on a technology race between black box machine learning models improved by post-hoc interpretation and intrinsic interpretable models boosted to gain accuracy. Finally, it discusses how concrete and promising hybrid approaches can provide pragmatic, short-term answers to banks and policy makers without swallowing up stakeholders with economical and ethical stakes in this technological race.

Defect Detection in Atomic Resolution Transmission Electron Microscopy Images Using Machine Learning

Point defects play a fundamental role in the discovery of new materials due to their strong influence on material properties and behavior. At present, imaging techniques based on transmission electron microscopy (TEM) are widely employed for characterizing point defects in materials. However, current methods for defect detection predominantly involve visual inspection of TEM images, which is laborious and poses difficulties in materials where defect related contrast is weak or ambiguous. Recent efforts to develop machine learning methods for the detection of point defects in TEM images have focused on supervised methods that require labeled training data that is generated via simulation. Motivated by a desire for machine learning methods that can be trained on experimental data, we propose two self-supervised machine learning algorithms that are trained solely on images that are defect-free. Our proposed methods use principal components analysis (PCA) and convolutional neural networks (CNN) to analyze a TEM image and predict the location of a defect. Using simulated TEM images, we show that PCA can be used to accurately locate point defects in the case where there is no imaging noise. In the case where there is imaging noise, we show that incorporating a CNN dramatically improves model performance. Our models rely on a novel approach that uses the residual between a TEM image and its PCA reconstruction.

Detecting Urban Events by Considering Long Temporal Dependency of Sentiment Strength in Geotagged Social Media Data

The development of location-based services facilitates the use of location data for detecting urban events. Currently, most studies based on location data model the pattern of an urban dynamic and then extract the anomalies, which deviate significantly from the pattern as urban events. However, few studies have considered the long temporal dependency of sentiment strength in geotagged social media data, and thus it is difficult to further improve the reliability of detection results. In this paper, we combined a sentiment analysis method and long short-term memory neural network for detecting urban events with geotagged social media data. We first applied a dictionary-based method to evaluate the positive and negative sentiment strength. Based on long short-term memory neural network, the long temporal dependency of sentiment strength in geotagged social media data was constructed. By considering the long temporal dependency, daily positive and negative sentiment strength are predicted. We extracted anomalies that deviated significantly from the prediction as urban events. For each event, event-related information was obtained by analyzing social media texts. Our results indicate that the proposed approach is a cost-effective way to detect urban events, such as festivals, COVID-19-related events and traffic jams. In addition, compared to existing methods, we found that accounting for a long temporal dependency of sentiment strength can significantly improve the reliability of event detection.

Nontechnical Losses Detection Through Coordinated BiWGAN and SVDD

Nontechnical losses (NTLs) are estimated to be considerable and increasing every year. Recently, high-resolution measurements from globally laid smart meters have brought deeper insights on users' consumption patterns that can be exploited potentially by NTL detection. However, consumption-pattern-based NTL detection is now facing two major challenges: the inefficiency of harnessing high dimensionality and the severe lack of fraudulent samples. To overcome them, an NTL detection model based on deep learning and anomaly detection is proposed in this article, namely bidirectional Wasserstein GAN and support vector data description-based NTL detector (BSBND). Motivated by the powerful ability of generative adversarial networks (GANs) to learn deep representation from high-dimensional distributions of data, in the BSBND, we utilized a BiWGAN for feature extraction from high-dimensional raw consumption records, and a one-class classifier trained only on benign samples-SVDD-is adopted to map features into judgments. Moreover, a novel alternate coordinating algorithm is proposed to optimize the cooperation between the upstream BiWGAN and the downstream SVDD, and also, an interpreting algorithm is proposed to visualize the basis of each fraudulent judgment. Case studies have demonstrated the superiority of the BSBND over the state of the arts, the powerful feature extraction ability of BiWGAN, and also the effectiveness of the proposed coordinating and interpreting algorithms.

Uncertain distance-based outlier detection with arbitrarily shaped data objects

Enabling information systems to face anomalies in the presence of uncertainty is a compelling and challenging task. In this work the problem of unsupervised outlier detection in large collections of data objects modeled by means of arbitrary multidimensional probability density functions is considered. We present a novel definition of uncertain distance-based outlier under the attribute level uncertainty model, according to which an uncertain object is an object that always exists but its actual value is modeled by a multivariate pdf. According to this definition an uncertain object is declared to be an outlier on the basis of the expected number of its neighbors in the dataset. To the best of our knowledge this is the first work that considers the unsupervised outlier detection problem on data objects modeled by means of arbitrarily shaped multidimensional distribution functions. We present the UDBOD algorithm which efficiently detects the outliers in an input uncertain dataset by taking advantages of three optimized phases, that are parameter estimation, candidate selection, and the candidate filtering. An experimental campaign is presented, including a sensitivity analysis, a study of the effectiveness of the technique, a comparison with related algorithms, also in presence of high dimensional data, and a discussion about the behavior of our technique in real case scenarios.

An Aberration Detection-Based Approach for Sentinel Syndromic Surveillance of COVID-19 and Other Novel Influenza-Like Illnesses

Coronavirus Disease 2019 (COVID-19) has emerged as a significant global concern, triggering harsh public health restrictions in a successful bid to curb its exponential growth. As discussion shifts towards relaxation of these restrictions, there is significant concern of second-wave resurgence. The key to managing these outbreaks is early detection and intervention, and yet there is significant lag time associated with usage of laboratory confirmed cases for surveillance purposes. To address this, syndromic surveillance can be considered to provide a timelier alternative for first-line screening. Existing syndromic surveillance solutions are however typically focused around a known disease and have limited capability to distinguish between outbreaks of individual diseases sharing similar syndromes. This poses a challenge for surveillance of COVID-19 as its active periods are tend to overlap temporally with other influenza-like illnesses. In this study we explore performing sentinel syndromic surveillance for COVID-19 and other influenza-like illnesses using a deep learning-based approach. Our methods are based on aberration detection utilizing autoencoders that leverages symptom prevalence distributions to distinguish outbreaks of two ongoing diseases that share similar syndromes, even if they occur concurrently. We first demonstrate that this approach works for detection of outbreaks of influenza, which has known temporal boundaries. We then demonstrate that the autoencoder can be trained to not alert on known and well-managed influenza-like illnesses such as the common cold and influenza. Finally, we applied our approach to 2019-2020 data in the context of a COVID-19 syndromic surveillance task to demonstrate how implementation of such a system could have provided early warning of an outbreak of a novel influenza-like illness that did not match the symptom prevalence profile of influenza and other known influenza-like illnesses.

Robust Classification of High-Dimensional Spectroscopy Data Using Deep Learning and Data Synthesis

This paper presents a new approach to classification of high-dimensional spectroscopy data and demonstrates that it outperforms other current state-of-the art approaches. The specific task we consider is identifying whether samples contain chlorinated solvents or not, based on their Raman spectra. We also examine robustness to classification of outlier samples that are not represented in the training set (negative outliers). A novel application of a locally connected neural network (NN) for the binary classification of spectroscopy data is proposed and demonstrated to yield improved accuracy over traditionally popular algorithms. Additionally, we present the ability to further increase the accuracy of the locally connected NN algorithm through the use of synthetic training spectra, and we investigate the use of autoencoder based one-class classifiers and outlier detectors. Finally, a two-step classification process is presented as an alternative to the binary and one-class classification paradigms. This process combines the locally connected NN classifier, the use of synthetic training data, and an autoencoder based outlier detector to produce a model which is shown to both produce high classification accuracy and be robust in the presence of negative outliers.

Intelligent Detection of Information Outliers Using Linguistic Summaries with Non-monotonic Quantifiers

In the processing of imprecise information, principally in big data analysis, it is very advantageous to transform numerical values into the standard form of linguistic statements. This paper deals with a novel method of outlier detection using linguistic summaries. Particular attention is devoted to examining the usefulness of non-monotonic quantifiers, which represent a fuzzy determination of the amount of analyzed data. The answer is positive. The use of non-monotonic quantifiers in the detection of outliers can provide a more significant value of the degree of truth of a linguistic summary. At the end, this paper provides a computational example of practical importance.

CloudDet: Interactive Visual Analysis of Anomalous Performances in Cloud Computing Systems

Detecting and analyzing potential anomalous performances in cloud computing systems is essential for avoiding losses to customers and ensuring the efficient operation of the systems. To this end, a variety of automated techniques have been developed to identify anomalies in cloud computing. These techniques are usually adopted to track the performance metrics of the system (e.g., CPU, memory, and disk I/O), represented by a multivariate time series. However, given the complex characteristics of cloud computing data, the effectiveness of these automated methods is affected. Thus, substantial human judgment on the automated analysis results is required for anomaly interpretation. In this paper, we present a unified visual analytics system named CloudDet to interactively detect, inspect, and diagnose anomalies in cloud computing systems. A novel unsupervised anomaly detection algorithm is developed to identify anomalies based on the specific temporal patterns of the given metrics data (e.g., the periodic pattern). Rich visualization and interaction designs are used to help understand the anomalies in the spatial and temporal context. We demonstrate the effectiveness of CloudDet through a quantitative evaluation, two case studies with real-world data, and interviews with domain experts.

Learning Neural Representations for Network Anomaly Detection

This paper proposes latent representation models for improving network anomaly detection. Well-known anomaly detection algorithms often suffer from challenges posed by network data, such as high dimension and sparsity, and a lack of anomaly data for training, model selection, and hyperparameter tuning. Our approach is to introduce new regularizers to a classical autoencoder (AE) and a variational AE, which force normal data into a very tight area centered at the origin in the nonsaturating area of the bottleneck unit activations. These trained AEs on normal data will push normal points toward the origin, whereas anomalies, which differ from normal data, will be put far away from the normal region. The models are very different from common regularized AEs, sparse AE, and contractive AE, in which the regularized AEs tend to make their latent representation less sensitive to changes of the input data. The bottleneck feature space is now used as a new data representation. A number of one-class learning algorithms are used for evaluating the proposed models. The experiments testify that our models help these classifiers to perform efficiently and consistently on high-dimensional and sparse network datasets, even with relatively few training points. More importantly, the models can minimize the effect of model selection on these classifiers since their performance is insensitive to a wide range of hyperparameter settings.

Cellular frustration algorithms for anomaly detection applications

Cellular frustrated models have been developed to describe how the adaptive immune system works. They are composed by independent agents that continuously pair and unpair depending on the information that one sub-set of these agents display. The emergent dynamics is sensitive to changes in the displayed information and can be used to detect anomalies, which can be important to accomplish the immune system main function of protecting the host. Therefore, it has been hypothesized that these models could be adequate to model the immune system activation. Likewise it has been hypothesized that these models could provide inspiration to develop new artificial intelligence algorithms for data mining applications. However, computational algorithms do not need to follow strictly the immunological reality. Here, we investigate efficient implementation strategies of these immune inspired ideas for anomaly detection applications and use real data to compare the performance of cellular frustration algorithms with standard implementations of one-class support vector machines and deep autoencoders. Our results demonstrate that more efficient implementations of cellular frustration algorithms are possible and also that cellular frustration algorithms can be advantageous for semi-supervised anomaly detection applications given their robustness and accuracy.

Neurodevelopmental heterogeneity and computational approaches for understanding autism

In recent years, the emerging field of computational psychiatry has impelled the use of machine learning models as a means to further understand the pathogenesis of multiple clinical disorders. In this paper, we discuss how autism spectrum disorder (ASD) was and continues to be diagnosed in the context of its complex neurodevelopmental heterogeneity. We review machine learning approaches to streamline ASD's diagnostic methods, to discern similarities and differences from comorbid diagnoses, and to follow developmentally variable outcomes. Both supervised machine learning models for classification outcome and unsupervised approaches to identify new dimensions and subgroups are discussed. We provide an illustrative example of how computational analytic methods and a longitudinal design can improve our inferential ability to detect early dysfunctional behaviors that may or may not reach threshold levels for formal diagnoses. Specifically, an unsupervised machine learning approach of anomaly detection is used to illustrate how community samples may be utilized to investigate early autism risk, multidimensional features, and outcome variables. Because ASD symptoms and challenges are not static within individuals across development, computational approaches present a promising method to elucidate subgroups of etiological contributions to phenotype, alternative developmental courses, interactions with biomedical comorbidities, and to predict potential responses to therapeutic interventions.

Security Evaluation of a Banking Fraud Analysis System

The significant growth of banking fraud, fueled by the underground economy of malware, has raised the need for effective detection systems. Therefore, in the last few years, banks have upgraded their security to protect transactions from fraud. State-of-the-art solutions detect fraud as deviations from customers’ spending habits. To the best of our knowledge, almost all existing approaches do not provide an in-depth model’s granularity and security analysis against elusive attacks.

In this article, we examine Banksealer, a decision support system for banking fraud analysis that evaluates the influence on detection performance of the granularity at which spending habits are modeled and its security against evasive attacks. First, we compare user-centric modeling, which builds a model for each user, with system-centric modeling, which builds a model for the entire system, from the point of view of detection performance. Then, we assess the robustness of Banksealer against malicious attackers that are aware of the structure of the models in use. To this end, we design and implement a proof-of-concept attack tool that performs mimicry attacks, emulating a sophisticated attacker that cloaks frauds to avoid detection. We experimentally confirm the feasibility of such attacks, their cost, and the effort required by an attacker in order to perform them. In addition, we discuss possible countermeasures.

We provide a comprehensive evaluation on a large real-world dataset obtained from one of the largest Italian banks.

EnsembleLens: Ensemble-based Visual Exploration of Anomaly Detection Algorithms with Multidimensional Data

The results of anomaly detection are sensitive to the choice of detection algorithms as they are specialized for different properties of data, especially for multidimensional data. Thus, it is vital to select the algorithm appropriately. To systematically select the algorithms, ensemble analysis techniques have been developed to support the assembly and comparison of heterogeneous algorithms. However, challenges remain due to the absence of the ground truth, interpretation, or evaluation of these anomaly detectors. In this paper, we present a visual analytics system named EnsembleLens that evaluates anomaly detection algorithms based on the ensemble analysis process. The system visualizes the ensemble processes and results by a set of novel visual designs and multiple coordinated contextual views to meet the requirements of correlation analysis, assessment and reasoning of anomaly detection algorithms. We also introduce an interactive analysis workflow that dynamically produces contextualized and interpretable data summaries that allow further refinements of exploration results based on user feedback. We demonstrate the effectiveness of EnsembleLens through a quantitative evaluation, three case studies with real-world data and interviews with two domain experts.

AnRAD: A Neuromorphic Anomaly Detection Framework for Massive Concurrent Data Streams

The evolution of high performance computing technologies has enabled the large-scale implementation of neuromorphic models and pushed the research in computational intelligence into a new era. Among the machine learning applications, unsupervised detection of anomalous streams is especially challenging due to the requirements of detection accuracy and real-time performance. Designing a computing framework that harnesses the growing computing power of the multicore systems while maintaining high sensitivity and specificity to the anomalies is an urgent research topic. In this paper, we propose anomaly recognition and detection (AnRAD), a bioinspired detection framework that performs probabilistic inferences. We analyze the feature dependency and develop a self-structuring method that learns an efficient confabulation network using unlabeled data. This network is capable of fast incremental learning, which continuously refines the knowledge base using streaming data. Compared with several existing anomaly detection approaches, our method provides competitive detection quality. Furthermore, we exploit the massive parallel structure of the AnRAD framework. Our implementations of the detection algorithm on the graphic processing unit and the Xeon Phi coprocessor both obtain substantial speedups over the sequential implementation on general-purpose microprocessor. The framework provides real-time service to concurrent data streams within diversified knowledge contexts, and can be applied to large problems with multiple local patterns. Experimental results demonstrate high computing performance and memory efficiency. For vehicle behavior detection, the framework is able to monitor up to 16000 vehicles (data streams) and their interactions in real time with a single commodity coprocessor, and uses less than 0.2 ms for one testing subject. Finally, the detection network is ported to our spiking neural network simulator to show the potential of adapting to the emerging neuromorphic architectures.