Distribution water quality anomaly detection from UV optical sensor monitoring data by integrating principal component analysis with chi-square distribution

Dynamic Multivariate Outlier Detection Algorithm Using Ultraviolet Visible Spectroscopy for Monitoring Surface Water Contamination With Hydrological Fluctuation in Real-Time

The contamination of surface water is of great harm. Ultraviolet-visible (UV-Vis) spectroscopy is an effective method to detect water contamination. However, surface water quality is influenced by hydrological fluctuation caused by rain, change of flow, etc., leading to changes of spectral characteristics over time. In the process of contamination detection, such changes cause confusion between hydrological fluctuation spectra and contaminated water spectra, thus increasing the false alarm rate. Besides, missing alarms of contaminated water is a common problem when the signal-to-noise ratio is low. In this paper, a dynamic multivariable outlier sampling rate detection (DM-SRD) algorithm is proposed. A dynamic updating strategy is introduced to increase adaptability to hydrological fluctuation. Additionally, multiple outlier variables are adopted as outlying degree indicators, which increases the accuracy of contamination detection. Two experiments were carried out using spectra collected from real surface water sites and hydrological fluctuation was constructed. To verify the effectiveness of the DM-SRD method, a comparison with the static SRD method and spectral match method was conducted. The results show that the accuracy of the DM-SRD method is 97.8%. Compared with the other two detection methods, DM-SRD significantly reduces false alarm rate and avoids missing alarms. Additionally, the results demonstrate that whether the database contained prior information on hydrological fluctuation or not, DM-SRD maintained high detection accuracy, which indicates great adaptability and robustness.

Fake news stance detection using selective features and FakeNET

The proliferation of fake news has severe effects on society and individuals on multiple fronts. With fast-paced online content generation, has come the challenging problem of fake news content. Consequently, automated systems to make a timely judgment of fake news have become the need of the hour. The performance of such systems heavily relies on feature engineering and requires an appropriate feature set to increase performance and robustness. In this context, this study employs two methods for reducing the number of feature dimensions including Chi-square and principal component analysis (PCA). These methods are employed with a hybrid neural network architecture of convolutional neural network (CNN) and long short-term memory (LSTM) model called FakeNET. The use of PCA and Chi-square aims at utilizing appropriate feature vectors for better performance and lower computational complexity. A multi-class dataset is used comprising 'agree', 'disagree', 'discuss', and 'unrelated' classes obtained from the Fake News Challenges (FNC) website. Further contextual features for identifying bogus news are obtained through PCA and Chi-Square, which are given nonlinear characteristics. The purpose of this study is to locate the article's perspective concerning the headline. The proposed approach yields gains of 0.04 in accuracy and 0.20 in the F1 score, respectively. As per the experimental results, PCA achieves a higher accuracy of 0.978 than both Chi-square and state-of-the-art approaches.

Applications of Online UV-Vis Spectrophotometer for Drinking Water Quality Monitoring and Process Control: A Review

Water quality monitoring is an essential component of water quality management for water utilities for managing the drinking water supply. Online UV-Vis spectrophotometers are becoming popular choices for online water quality monitoring and process control, as they are reagent free, do not require sample pre-treatments and can provide continuous measurements. The advantages of the online UV-Vis sensors are that they can capture events and allow quicker responses to water quality changes compared to conventional water quality monitoring. This review summarizes the applications of online UV-Vis spectrophotometers for drinking water quality management in the last two decades. Water quality measurements can be performed directly using the built-in generic algorithms of the online UV-Vis instruments, including absorbance at 254 nm (UV254), colour, dissolved organic carbon (DOC), total organic carbon (TOC), turbidity and nitrate. To enhance the usability of this technique by providing a higher level of operations intelligence, the UV-Vis spectra combined with chemometrics approach offers simplicity, flexibility and applicability. The use of anomaly detection and an early warning was also discussed for drinking water quality monitoring at the source or in the distribution system. As most of the online UV-Vis instruments studies in the drinking water field were conducted at the laboratory- and pilot-scale, future work is needed for industrial-scale evaluation with ab appropriate validation methodology. Issues and potential solutions associated with online instruments for water quality monitoring have been provided. Current technique development outcomes indicate that future research and development work is needed for the integration of early warnings and real-time water treatment process control systems using the online UV-Vis spectrophotometers as part of the water quality management system.

Using machine learning classification to detect simulated increases of de facto reuse and urban stormwater surges in surface water

Water quality events such as increases in stormwater or wastewater effluent in drinking water sources pose hazards to drinking water consumers. Stormwater and wastewater effluent enter Lake Mead-an important drinking water source in the southwest USA-via the Las Vegas Wash. Previous studies have applied machine learning and online instruments to detect contamination in water distribution systems. However, alert systems at drinking water intakes would provide more time for corrective action. An array of online instruments measuring pH, conductivity, redox potential, turbidity, temperature, tryptophan-like fluorescence, UV absorbance (UVA₂₅₄), TOC, and chlorophyll-a was fed raw water directly from Lake Mead. Wastewater effluent, dry weather Las Vegas Wash, and storm-impacted Las Vegas Wash samples were blended into the instrument inlets at known ratios to simulate three types of adverse water quality events. Data preprocessing was conducted to correct for diurnal patterns or instrument drift. Supervised machine learning was conducted using previously published models in R. Ninety-nine models were screened on the raw data. Eight high-performing models were evaluated in-depth and optimized. Weighted k-Nearest Neighbors, Single C5.0 Ruleset, Mixture Discriminant Analysis, and an ensemble of these three models had accuracy over 97% when assigning test set data among three classes (Normal, Event, or Maintenance). The ensemble detected all event types at the earliest timepoint and had one false positive that was not a lag error (i.e., consecutively following a true positive). Omitting Maintenance, the Adaboost model had over 99% test set accuracy and zero false positives that were not lag errors. Data preprocessing was beneficial, but the optimal methods were model-specific. All nine water quality variables were useful for most models, but UVA₂₅₄ and turbidity were most important.

Advances on Water Quality Detection by UV-Vis Spectroscopy

Water resources are closely linked to human productivity and life. Owing to the deteriorating water resources environment, accurate and rapid determination of the main water quality parameters has become a current research hotspot. Ultraviolet-visible (UV-Vis) spectroscopy offers an effective tool for qualitative analysis and quantitative detection of contaminants in a water environment. In this review, the principle and application of UV-Vis technology in water quality detection were studied. The principle of UV-Vis spectroscopy for detecting water quality parameters and the method of modeling and analysis of spectral data were presented. Various UV-Vis technologies for water quality detection were reviewed according to the types of pollutants, such as chemical oxygen demand, heavy metal ions, nitrate nitrogen, and dissolved organic carbon. Finally, the future development of UV-Vis spectroscopy for the determination of water quality was discussed.

Classification of water contamination developed by 2-D Gabor wavelet analysis and support vector machine based on fluorescence spectroscopy

The identification of the specific categories of pollutants in the urban water supply system is necessary. Traditional detection methods are based mainly on common water quality indicators. However, inspecting these water quality indicators is made difficult by issues such as long analysis time, insufficient sensitivity, need for reagents, and generation of waste liquid. These problems hinder high-frequency water detection and monitoring. In this study, three-dimensional (3D) fluorescence spectroscopy is adopted as a monitoring method for water quality. An identification method based on two-dimensional (2D) Gabor wavelets and support vector machine (SVM) multi-classification is also proposed. The Delaunay triangulation method for interpolation is used to pre-process 3D fluorescence spectra and thereby eliminate Rayleigh scattering and Raman scattering. A 2D Gabor wavelet function generated by filters of different scales and rotation angles is proposed to extract the features of the spectra. The block statistics method, based on Gabor feature description, is employed to enhance the efficiency in describing spectra features. Then, multiple SVM classifiers are used in pollutant classification and recognition. By comparing the proposed method with principal component analysis, which is a commonly used feature extraction method, this study finds that the application of 2D Gabor wavelets and block statistics can effectively describe the characteristics of 3D fluorescence spectra. Moreover, 2D Gabor wavelets achieve high classification accuracy, especially for substances with closely positioned or overlapping characteristic peaks.

Adaptive Detection Method for Organic Contamination Events in Water Distribution Systems Using the UV-Vis Spectrum Based on Semi-Supervised Learning

A method that uses the ultraviolet-visible (UV-Vis) spectrum to detect organic contamination events in water distribution systems exhibits the advantages of rapid detection, low cost, and no need for reagents. The speed, accuracy, and comprehensive analysis of such a method meet the requirements for online water quality monitoring. However, the UV-Vis spectrum is easily disturbed by environmental factors that cause fluctuations of the spectrum and result in false alarms. This study proposes an adaptive method for detecting organic contamination events in water distribution systems that uses the UV-Vis spectrum based on a semi-supervised learning model. This method modifies the baseline using dynamic orthogonal projection correction and adjusts the support vector regression model in real time. Thus, an adaptive online anomaly detection model that maximizes the use of unlabeled data is obtained. Experimental results demonstrate that the proposed method is adaptive to baseline drift and exhibits good performance in detecting organic contamination events in water distribution systems.

In situ detection of water quality contamination events based on signal complexity analysis using online ultraviolet-visible spectral sensor

The contaminant detection in water distribution systems is essential to protect public health from potentially harmful compounds resulting from accidental spills or intentional releases. As a noninvasive optical technique, ultraviolet-visible (UV-Vis) spectroscopy is investigated for detecting contamination events. However, current methods for event detection exhibit the shortcomings of noise susceptibility. In this paper, a new method that has less sensitivity to noise was proposed to detect water quality contamination events by analyzing the complexity of the UV-Vis spectrum series. The proposed method applied approximate entropy (ApEn) to measure spectrum signals' complexity, which made a distinction between normal and abnormal signals. The impact of noise was attenuated with the help of ApEn's insensitivity to signal disturbance. This method was tested on a real water distribution system data set with various concentration simulation events. Results from the experiment and analysis show that the proposed method has a good performance on noise tolerance and provides a better detection result compared with the autoregressive model and sequential probability ratio test.