Towards Near-Real-Time Intrusion Detection for IoT Devices using Supervised Learning and Apache Spark

An efficient intrusion detection model based on convolutional spiking neural network

Many intrusion detection techniques have been developed to ensure that the target system can function properly under the established rules. With the booming Internet of Things (IoT) applications, the resource-constrained nature of its devices makes it urgent to explore lightweight and high-performance intrusion detection models. Recent years have seen a particularly active application of deep learning (DL) techniques. The spiking neural network (SNN), a type of artificial intelligence that is associated with sparse computations and inherent temporal dynamics, has been viewed as a potential candidate for the next generation of DL. It should be noted, however, that current research into SNNs has largely focused on scenarios where limited computational resources and insufficient power sources are not considered. Consequently, even state-of-the-art SNN solutions tend to be inefficient. In this paper, a lightweight and effective detection model is proposed. With the help of rational algorithm design, the model integrates the advantages of SNNs as well as convolutional neural networks (CNNs). In addition to reducing resource usage, it maintains a high level of classification accuracy. The proposed model was evaluated against some current state-of-the-art models using a comprehensive set of metrics. Based on the experimental results, the model demonstrated improved adaptability to environments with limited computational resources and energy sources.

A distributed computing model for big data anonymization in the networks

Recently big data and its applications had sharp growth in various fields such as IoT, bioinformatics, eCommerce, and social media. The huge volume of data incurred enormous challenges to the architecture, infrastructure, and computing capacity of IT systems. Therefore, the compelling need of the scientific and industrial community is large-scale and robust computing systems. Since one of the characteristics of big data is value, data should be published for analysts to extract useful patterns from them. However, data publishing may lead to the disclosure of individuals' private information. Among the modern parallel computing platforms, Apache Spark is a fast and in-memory computing framework for large-scale data processing that provides high scalability by introducing the resilient distributed dataset (RDDs). In terms of performance, Due to in-memory computations, it is 100 times faster than Hadoop. Therefore, Apache Spark is one of the essential frameworks to implement distributed methods for privacy-preserving in big data publishing (PPBDP). This paper uses the RDD programming of Apache Spark to propose an efficient parallel implementation of a new computing model for big data anonymization. This computing model has three-phase of in-memory computations to address the runtime, scalability, and performance of large-scale data anonymization. The model supports partition-based data clustering algorithms to preserve the λ-diversity privacy model by using transformation and actions on RDDs. Therefore, the authors have investigated Spark-based implementation for preserving the λ-diversity privacy model by two designed City block and Pearson distance functions. The results of the paper provide a comprehensive guideline allowing the researchers to apply Apache Spark in their own researches.

Detecting Reconnaissance and Discovery Tactics from the MITRE ATT&CK Framework in Zeek Conn Logs Using Spark's Machine Learning in the Big Data Framework

While computer networks and the massive amount of communication taking place on these networks grow, the amount of damage that can be done by network intrusions grows in tandem. The need is for an effective and scalable intrusion detection system (IDS) to address these potential damages that come with the growth of these networks. A great deal of contemporary research on near real-time IDS focuses on applying machine learning classifiers to labeled network intrusion datasets, but these datasets need be relevant pertaining to the currency of the network intrusions. This paper focuses on a newly created dataset, UWF-ZeekData22, that analyzes data from Zeek's Connection Logs collected using Security Onion 2 network security monitor and labelled using the MITRE ATT&CK framework TTPs. Due to the volume of data, Spark, in the big data framework, was used to run many of the well-known classifiers (naïve Bayes, random forest, decision tree, support vector classifier, gradient boosted trees, and logistic regression) to classify the reconnaissance and discovery tactics from this dataset. In addition to looking at the performance of these classifiers using Spark, scalability and response time were also analyzed.

Towards Developing a Robust Intrusion Detection Model Using Hadoop-Spark and Data Augmentation for IoT Networks

In recent years, anomaly detection and machine learning for intrusion detection systems have been used to detect anomalies on Internet of Things networks. These systems rely on machine and deep learning to improve the detection accuracy. However, the robustness of the model depends on the number of datasamples available, quality of the data, and the distribution of the data classes. In the present paper, we focused specifically on the amount of data and class imbalanced since both parameters are key in IoT due to the fact that network traffic is increasing exponentially. For this reason, we propose a framework that uses a big data methodology with Hadoop-Spark to train and test multi-class and binary classification with one-vs-rest strategy for intrusion detection using the entire BoT IoT dataset. Thus, we evaluate all the algorithms available in Hadoop-Spark in terms of accuracy and processing time. In addition, since the BoT IoT dataset used is highly imbalanced, we also improve the accuracy for detecting minority classes by generating more datasamples using a Conditional Tabular Generative Adversarial Network (CTGAN). In general, our proposed model outperforms other published models including our previous model. Using our proposed methodology, the F1-score of one of the minority class, i.e., Theft attack was improved from 42% to 99%.

Fuzzy Local Information and Bhattacharya-Based C-Means Clustering and Optimized Deep Learning in Spark Framework for Intrusion Detection

Strong network connections make the risk of malicious activities emerge faster while dealing with big data. An intrusion detection system (IDS) can be utilized for alerting suitable entities when hazardous actions are occurring. Most of the techniques used to classify intrusions lack the techniques executed with big data. This paper devised an optimization-driven deep learning technique for detecting the intrusion using the Spark model. The input data is fed to the data partitioning phase wherein the partitioning of data is done using the proposed fuzzy local information and Bhattacharya-based C-means (FLIBCM). The proposed FLIBCM was devised by combining Bhattacharya distance and fuzzy local information C-Means (FLICM). The feature selection was achieved with classwise info gained to select imperative features. The data augmentation was done with oversampling to make it apposite for further processing. The detection of intrusion was done using a deep Maxout network (DMN), which was trained using the proposed student psychology water cycle caviar (SPWCC) obtained by combining the water cycle algorithm (WCA), the conditional autoregressive value at risk by regression quantiles (CAViaR), and the student psychology-based optimization algorithm (SPBO). The proposed SPWCC-based DMN offered enhanced performance with the highest accuracy of 97.6%, sensitivity of 98%, and specificity of 97%.

Learning-Based Methods for Cyber Attacks Detection in IoT Systems: Methods, Analysis, and Future Prospects

Internet of Things (IoT) is a developing technology that provides the simplicity and benefits of exchanging data with other devices using the cloud or wireless networks. However, the changes and developments in the IoT environment are making IoT systems susceptible to cyber attacks which could possibly lead to malicious intrusions. The impacts of these intrusions could lead to physical and economical damages. This article primarily focuses on the IoT system/framework, the IoT, learning-based methods, and the difficulties faced by the IoT devices or systems after the occurrence of an attack. Learning-based methods are reviewed using different types of cyber attacks, such as denial-of-service (DoS), distributed denial-of-service (DDoS), probing, user-to-root (U2R), remote-to-local (R2L), botnet attack, spoofing, and man-in-the-middle (MITM) attacks. For learning-based methods, both machine and deep learning methods are presented and analyzed in relation to the detection of cyber attacks in IoT systems. A comprehensive list of publications to date in the literature is integrated to present a complete picture of various developments in this area. Finally, future research directions are also provided in the paper.

A Review of Efficient Real-Time Decision Making in the Internet of Things

Emerging applications of IoT (the Internet of Things), such as smart transportation, health, and energy, are envisioned to greatly enhance the societal infrastructure and quality of life of individuals. In such innovative IoT applications, cost-efficient real-time decision-making is critical to facilitate, for example, effective transportation management and healthcare. In this paper, we formally define real-time decision tasks in IoT, review cutting-edge approaches that aim to efficiently schedule real-time decision tasks to meet their timing and data freshness constraints, review state-of-the-art approaches for efficient sensor data analytics in IoT, and discuss future research directions.

A hybrid machine learning approach for detecting unprecedented DDoS attacks

Service availability plays a vital role on computer networks, against which Distributed Denial of Service (DDoS) attacks are an increasingly growing threat each year. Machine learning (ML) is a promising approach widely used for DDoS detection, which obtains satisfactory results for pre-known attacks. However, they are almost incapable of detecting unknown malicious traffic. This paper proposes a novel method combining both supervised and unsupervised algorithms. First, a clustering algorithm separates the anomalous traffic from the normal data using several flow-based features. Then, using certain statistical measures, a classification algorithm is used to label the clusters. Employing a big data processing framework, we evaluate the proposed method by training on the CICIDS2017 dataset and testing on a different set of attacks provided in the more up-to-date CICDDoS2019. The results demonstrate that the Positive Likelihood Ratio (LR+) of our method is approximately 198% higher than the ML classification algorithms.

A Novel Approach for Network Intrusion Detection Using Multistage Deep Learning Image Recognition

The current rise in hacking and computer network attacks throughout the world has heightened the demand for improved intrusion detection and prevention solutions. The intrusion detection system (IDS) is critical in identifying abnormalities and assaults on the network, which have grown in size and pervasiveness. The paper proposes a novel approach for network intrusion detection using multistage deep learning image recognition. The network features are transformed into four-channel (Red, Green, Blue, and Alpha) images. The images then are used for classification to train and test the pre-trained deep learning model ResNet50. The proposed approach is evaluated using two publicly available benchmark datasets, UNSW-NB15 and BOUN Ddos. On the UNSW-NB15 dataset, the proposed approach achieves 99.8% accuracy in the detection of the generic attack. On the BOUN DDos dataset, the suggested approach achieves 99.7% accuracy in the detection of the DDos attack and 99.7% accuracy in the detection of the normal traffic.

A Recommendation Engine for Predicting Movie Ratings Using a Big Data Approach

In this era of big data, the amount of video content has dramatically increased with an exponential broadening of video streaming services. Hence, it has become very strenuous for end-users to search for their desired videos. Therefore, to attain an accurate and robust clustering of information, a hybrid algorithm was used to introduce a recommender engine with collaborative filtering using Apache Spark and machine learning (ML) libraries. In this study, we implemented a movie recommendation system based on a collaborative filtering approach using the alternating least squared (ALS) model to predict the best-rated movies. Our proposed system uses the last search data of a user regarding movie category and references this to instruct the recommender engine, thereby making a list of predictions for top ratings. The proposed study used a model-based approach of matrix factorization, the ALS algorithm along with a collaborative filtering technique, which solved the cold start, sparse, and scalability problems. In particular, we performed experimental analysis and successfully obtained minimum root mean squared errors (oRMSEs) of 0.8959 to 0.97613, approximately. Moreover, our proposed movie recommendation system showed an accuracy of 97% and predicted the top 1000 ratings for movies.

Anomaly Detection for Individual Sequences with Applications in Identifying Malicious Tools

Anomaly detection refers to the problem of identifying abnormal behaviour within a set of measurements. In many cases, one has some statistical model for normal data, and wishes to identify whether new data fit the model or not. However, in others, while there are normal data to learn from, there is no statistical model for this data, and there is no structured parameter set to estimate. Thus, one is forced to assume an individual sequences setup, where there is no given model or any guarantee that such a model exists. In this work, we propose a universal anomaly detection algorithm for one-dimensional time series that is able to learn the normal behaviour of systems and alert for abnormalities, without assuming anything on the normal data, or anything on the anomalies. The suggested method utilizes new information measures that were derived from the Lempel–Ziv (LZ) compression algorithm in order to optimally and efficiently learn the normal behaviour (during learning), and then estimate the likelihood of new data (during operation) and classify it accordingly. We apply the algorithm to key problems in computer security, as well as a benchmark anomaly detection data set, all using simple, single-feature time-indexed data. The first is detecting Botnets Command and Control (C&C) channels without deep inspection. We then apply it to the problems of malicious tools detection via system calls monitoring and data leakage identification.We conclude with the New York City (NYC) taxi data. Finally, while using information theoretic tools, we show that an attacker’s attempt to maliciously fool the detection system by trying to generate normal data is bound to fail, either due to a high probability of error or because of the need for huge amounts of resources.