Use of spectra in the visible and near-mid-ultraviolet range with principal component analysis and partial least squares processing for monitoring of suspended solids in municipal wastewater treatment plants

Optimization of indirect wastewater characterization using led spectrophotometry: a comparative analysis of regression, scaling, and dimensionality reduction methods

LED spectrophotometry is a robust technique for the indirect characterization of wastewater pollutant load through correlation modeling. To tackle this issue, a dataset with 1300 samples was collected, from both raw and treated wastewater from 45 wastewater treatment plants in Spain and Chile collected over 4 years. The type of regressor, scaling, and dimensionality reduction technique and nature of the data play crucial roles in the performance of the processing pipeline. Eighty-four pipelines were tested through exhaustive experimentation resulting from the combination of 7 regression techniques, 3 scaling methods, and 4 possible dimensional reductions. Those combinations were tested on the prediction of chemical oxygen demand (COD) and total suspended solids (TSS). Each pipeline underwent a tenfold cross-validation on 15 sub-datasets derived from the original dataset, accounting for variations in plants and wastewater types. The results point to the normalization of the data followed by a conversion through the PCA to finally apply a Random Forest Regressor as the combination which stood out These results highlight the importance of modeling strategies in wastewater management using techniques such as LED spectrophotometry.

A Design of Real-Time Data Acquisition and Processing System for Nanosecond Ultraviolet-Visible Absorption Spectrum Detection

Ultraviolet-visible absorption spectroscopy is widely used to monitor water quality, and rapid optical signal detection is a key technology in the process of spectrum measurement. In this paper, an ultrafast spectrophotometer system that can achieve spectrum data acquisition in a single flash of the xenon lamp (within 200 ns) is introduced, and a real-time denoising method for the spectrum is implemented on a field programmable gate array (FPGA) to work cooperatively with the nanosecond spectrum acquisition system, in order to guarantee the quality of the spectrum signals without losing running speed. The hardware of the data acquisition and processing system are constructed on a Xilinx Spartan 6 FPGA chip and its peripheral circuit, including an analog to digital converter and a complementary metal-oxide-semiconductor transistor (CMOS) sensor’s diver circuit. An oversampling method that is suitable for the CMOS sensor’s output is proposed, which works on the CMOS sensor’s dark current noise and readout noise. Another moving-average filter method is designed adaptively, which works on the low-frequency component to filter out the residual spectrum noise of the spectrum signal. The implementation of the filter on the FPGA has been optimized by using a pipelined structure and dual high-speed random-access memory (RAM). As a result, the CMOS linear image sensor successfully captured the spectrum of xenon flash light at the readout clock frequency of 500 kHz and the processing manipulation to the full UV-Vis spectrum data was accomplished at a sub-microsecond speed performance. After the digital filter and oversampling technology were implemented, the coefficient of variation of the measurements reduced from 9.57% to 1.74%, while the signal noise ratio (SNR) of the absorption spectrum increased nine times, compared to the raw data of the CMOS sensor’s output. The tests towards different analyte samples were conducted, and the system shows good performance on distinguishing different concentrations of different analyte solutions on both ultra-violet and visible spectrum bands. The present work showcases the potential of the CMOS sensor’s technique for the fast detection of contaminated water containing nitrate and organic compounds.

In situ detection of water quality contamination events based on signal complexity analysis using online ultraviolet-visible spectral sensor

The contaminant detection in water distribution systems is essential to protect public health from potentially harmful compounds resulting from accidental spills or intentional releases. As a noninvasive optical technique, ultraviolet-visible (UV-Vis) spectroscopy is investigated for detecting contamination events. However, current methods for event detection exhibit the shortcomings of noise susceptibility. In this paper, a new method that has less sensitivity to noise was proposed to detect water quality contamination events by analyzing the complexity of the UV-Vis spectrum series. The proposed method applied approximate entropy (ApEn) to measure spectrum signals' complexity, which made a distinction between normal and abnormal signals. The impact of noise was attenuated with the help of ApEn's insensitivity to signal disturbance. This method was tested on a real water distribution system data set with various concentration simulation events. Results from the experiment and analysis show that the proposed method has a good performance on noise tolerance and provides a better detection result compared with the autoregressive model and sequential probability ratio test.

Real-time detection of organic contamination events in water distribution systems by principal components analysis of ultraviolet spectral data

The detection of organic contaminants in water distribution systems is essential to protect public health from potential harmful compounds resulting from accidental spills or intentional releases. Existing methods for detecting organic contaminants are based on quantitative analyses such as chemical testing and gas/liquid chromatography, which are time- and reagent-consuming and involve costly maintenance. This study proposes a novel procedure based on discrete wavelet transform and principal component analysis for detecting organic contamination events from ultraviolet spectral data. Firstly, the spectrum of each observation is transformed using discrete wavelet with a coiflet mother wavelet to capture the abrupt change along the wavelength. Principal component analysis is then employed to approximate the spectra based on capture and fusion features. The significant value of Hotelling's T² statistics is calculated and used to detect outliers. An alarm of contamination event is triggered by sequential Bayesian analysis when the outliers appear continuously in several observations. The effectiveness of the proposed procedure is tested on-line using a pilot-scale setup and experimental data.

Calibration Transfer Between a Bench Scanning and a Submersible Diode Array Spectrophotometer for In Situ Wastewater Quality Monitoring in Sewer Systems

Online monitoring programs based on spectroscopy have a high application potential for the detection of hazardous wastewater discharges in sewer systems. Wastewater hydraulics poses a challenge for in situ spectroscopy, especially when the system includes storm water connections leading to rapid changes in water depth, velocity, and in the water quality matrix. Thus, there is a need to optimize and fix the location of in situ instruments, limiting their availability for calibration. In this context, the development of calibration models on bench spectrophotometers to estimate wastewater quality parameters from spectra acquired with in situ instruments could be very useful. However, spectra contain information not only from the samples, but also from the spectrophotometer generally invalidating this approach. The use of calibration transfer methods is a promising solution to this problem. In this study, calibration models were developed using interval partial least squares (iPLS), for the estimation of total suspended solids (TSS) and chemical oxygen demand (COD) in sewage from Ultraviolet-visible spectra acquired in a bench scanning spectrophotometer. The feasibility of calibration transfer to a submersible, diode array equipment, to be subsequently operated in situ, was assessed using three procedures: slope and bias correction (SBC); single wavelength standardization (SWS) on mean spectra; and local centering (LC). The results showed that SBC was the most adequate for the available data, adding insignificant error to the base model estimates. Single wavelength standardization was a close second best, potentially more robust, and independent of the base iPLS model. Local centering was shown to be inadequate for the samples and instruments used.

Distribution water quality anomaly detection from UV optical sensor monitoring data by integrating principal component analysis with chi-square distribution

The issue of distribution water quality security ensuring is recently attracting global attention due to the potential threat from harmful contaminants. The real-time monitoring based on ultraviolet optical sensors is a promising technique. This method is of reagent-free, low maintenance cost, rapid analysis and wide cover range. However, the ultraviolet absorption spectra are of large size and easily interfered. While within the on-site application, there is almost no prior knowledge like spectral characteristics of potential contaminants before determined. Meanwhile, the concept of normal water quality is also varying due to the operating condition. In this paper, a procedure based on multivariate statistical analysis is proposed to detect distribution water quality anomaly based on ultraviolet optical sensors. Firstly, the principal component analysis is employed to capture the main variety features from the spectral matrix and reduce the dimensionality. A new statistical variable is then constructed and used for evaluating the local outlying degree according to the chi-square distribution in the principal component subspace. The possibility of anomaly of the latest observation is calculated by the accumulation of the outlying degrees from the adjacent previous observations. To develop a more reliable anomaly detection procedure, several key parameters are discussed. By utilizing the proposed methods, the distribution water quality anomalies and the optical abnormal changes can be detected. The contaminants intrusion experiment is conducted in a pilot-scale distribution system by injecting phenol solution. The effectiveness of the proposed procedure is finally testified using the experimental spectral data.