A three-stage deep learning-based training frame for spectra baseline correction

For spectrometers, baseline drift seriously affects the measurement and quantitative analysis of spectral data. Deep learning has recently emerged as a powerful method for baseline correction. However, the dependence on vast amounts of paired data and the difficulty in obtaining spectral data limit the performance and development of deep learning-based methods. Therefore, we solve these problems from the network architecture and training framework. For the network architecture, a Learned Feature Fusion (LFF) module is designed to improve the performance of U-net, and a three-stage training frame is proposed to train this network. Specifically, the LFF module is designed to adaptively integrate features from different scales, greatly improving the performance of U-net. For the training frame, stage 1 uses airPLS to ameliorate the problem of vast amounts of paired data, stage 2 uses synthetic spectra to further ease reliance on real spectra, and stage 3 uses contrastive learning to reduce the gap between synthesized and real spectra. The experiments show that the proposed method is a powerful tool for baseline correction and possesses potential for application in spectral quantitative analysis.

Baseline correction for Raman spectra using a spectral estimation-based asymmetrically reweighted penalized least squares method

Baseline correction is necessary for the qualitative and quantitative analysis of samples because of the existence of background fluorescence interference in Raman spectra. The asymmetric least squares (ALS) method is an adaptive and automated algorithm that avoids peak detection operations along with other user interactions. However, current ALS-based improved algorithms only consider the smoothness configuration of regions where the signals are greater than the fitted baseline, which results in smoothing distortion. In this paper, an asymmetrically reweighted penalized least squares method based on spectral estimation (SEALS) is proposed. SEALS considers not only the uniform distribution of additive noise along the baseline but also the energy distribution of the signal above and below the fitted baseline. The energy distribution is estimated using inverse Fourier and autoregressive models to create a spectral estimation kernel. This kernel effectively optimizes and balances the asymmetric weight assigned to each data point. By doing so, it resolves the issue of local oversmoothing that is typically encountered in the asymmetrically reweighted penalized least squares method. This oversmoothing problem can negatively impact the iteration depth and accuracy of baseline fitting. In comparative experiments on simulated spectra, SEALS demonstrated a better baseline fitting performance compared to several other advanced baseline correction methods, both under moderate and strong fluorescence backgrounds. It has also been proven to be highly resistant to noise interference. When applied to real Raman spectra, the algorithm correctly restored the weak peaks and removed the fluorescence peaks, demonstrating the effectiveness of this method. The computation time of the proposed method was approximately 0.05 s, which satisfies the real-time baseline correction requirements of practical spectroscopy acquisition.

An improved PD-AsLS method for baseline estimation in EDXRF analysis

Baseline correction is an important step in energy-dispersive X-ray fluorescence analysis. The asymmetric least squares method (AsLS), adaptive iteratively reweighted penalized least squares method (airPLS), and asymmetrically reweighted penalized least squares method (arPLS) are widely used to automatically select the data points for the baseline. Considering the parametric sensitivity of the aforementioned methods and the statistical characteristics of the X-ray energy spectrum, this paper proposes an asymmetrically reweighted penalized least squares method based on the Poisson distribution (PD-AsLS) to automatically correct the baseline of X-ray spectra. Monte Carlo (MC) simulation is used to obtain the background spectrum, and PD-AsLS is used to estimate the baseline of the background. The relative error and the absolute error between the simulated background and PD-AsLS estimated background are used to determine the accuracy of PD-AsLS. The correlation coefficient (COR) and the root mean square error (RMSE) between the estmated baseline and the real baseline are calculated, and results of PD-AsLS are compared with results of three other classical methods (arPLS, airPLS and AsLS) to evaluate the reliability of PD-AsLS. The results of PD-AsLS show that the COR is above 0.95 and RMSE is less than 6. The stability and the practicability of PD-AsLS are also evaluated in experiments. A sample is measured five time to get its X-ray energy spectra, and the coefficient of variation (CV) of the estimated baseline is smaller than that of measured spectra. Experiments show that PD-AsLS can estimate baselines better than arPLS without any overestimation. Those results indicate that PD-AsLS can reliably estimate the baselines of X-ray spectra and effectively suppress the statistical fluctuation.

Critical Review of Processing and Classification Techniques for Images and Spectra in Microplastic Research

Microplastic research is a rapidly developing field, with urgent needs for high throughput and automated analysis techniques. We conducted a review covering image analysis from optical microscopy, scanning electron microscopy, fluorescence microscopy, and spectral analysis from Fourier transform infrared (FT-IR) spectroscopy, Raman spectroscopy, pyrolysis gas-chromatography mass-spectrometry, and energy dispersive X-ray spectroscopy. These techniques were commonly used to collect, process, and interpret data from microplastic samples. This review outlined and critiques current approaches for analysis steps in image processing (color, thresholding, particle quantification), spectral processing (background and baseline subtraction, smoothing and noise reduction, data transformation), image classification (reference libraries, morphology, color, and fluorescence intensity), and spectral classification (reference libraries, matching procedures, and best practices for developing in-house reference tools). We highlighted opportunities to advance microplastic data analysis and interpretation by (i) quantifying colors, shapes, sizes, and surface topologies with image analysis software, (ii) identifying threshold values of particle characteristics in images that distinguish plastic particles from other particles, (iii) advancing spectral processing and classification routines, (iv) creating and sharing robust spectral libraries, (v) conducting double blind and negative controls, (vi) sharing raw data and analysis code, and (vii) leveraging readily available data to develop machine learning classification models. We identified analytical needs that we could fill and developed supplementary information for a reference library of plastic images and spectra, a tutorial for basic image analysis, and a code to download images from peer reviewed literature. Our major findings were that research on microplastics was progressing toward the use of multiple analytical methods and increasingly incorporating chemical classification. We suggest that new and repurposed methods need to be developed for high throughput screening using a diversity of approaches and highlight machine learning as one potential avenue toward this capability.

Recent applications of chemometrics in one- and two-dimensional chromatography

The proliferation of increasingly more sophisticated analytical separation systems, often incorporating increasingly more powerful detection techniques, such as high-resolution mass spectrometry, causes an urgent need for highly efficient data-analysis and optimization strategies. This is especially true for comprehensive two-dimensional chromatography applied to the separation of very complex samples. In this contribution, the requirement for chemometric tools is explained and the latest developments in approaches for (pre-)processing and analyzing data arising from one- and two-dimensional chromatography systems are reviewed. The final part of this review focuses on the application of chemometrics for method development and optimization.

An Automatic Baseline Correction Method Based on the Penalized Least Squares Method

Baseline drift spectra are used for quantitative and qualitative analysis, which can easily lead to inaccurate or even wrong results. Although there are several baseline correction methods based on penalized least squares, they all have one or more parameters that must be optimized by users. For this purpose, an automatic baseline correction method based on penalized least squares is proposed in this paper. The algorithm first linearly expands the ends of the spectrum signal, and a Gaussian peak is added to the expanded range. Then, the whole spectrum is corrected by the adaptive smoothness parameter penalized least squares (asPLS) method, that is, by turning the smoothing parameter λ of asPLS to obtain a different root-mean-square error (RMSE) in the extended range, the optimal λ is selected with minimal RMSE. Finally, the baseline of the original signal is well estimated by asPLS with the optimal λ. The paper concludes with the experimental results on the simulated spectra and measured infrared spectra, demonstrating that the proposed method can automatically deal with different types of baseline drift.

A Statistical Approach of Background Removal and Spectrum Identification for SERS Data

SERS (surface-enhanced Raman scattering) enhances the Raman signals, but the plasmonic effects are sensitive to the chemical environment and the coupling between nanoparticles, resulting in large and variable backgrounds, which make signal matching and analyte identification highly challenging. Removing background is essential, but existing methods either cannot fit the strong fluctuation of the SERS spectrum or do not consider the spectra’s shape change across time. Here we present a new statistical approach named SABARSI that overcomes these difficulties by combining information from multiple spectra. Further, after efficiently removing the background, we have developed the first automatic method, as a part of SABARSI, for detecting signals of molecules and matching signals corresponding to identical molecules. The superior efficiency and reproducibility of SABARSI are shown on two types of experimental datasets.

Raman Spectroscopy in Nonwoody Plants

Confocal Raman spectroscopy (RS) enables obtaining molecular information from the nondestructive analysis of plant material in situ. It can thereby be a useful method to investigate spatial distribution and heterogeneity of cell-wall polymers. The authors' intention is to present some examples of RS application and its capabilities for investigations of nonwoody plants. In this context, we present protocols for qualitative analysis of main polymers of plant wall and application of RS in a semiquantitative study of the arrangement of selected polymers in the wall in its native state.

Study on baseline correction methods for the Fourier transform infrared spectra with different signal-to-noise ratios

Removing the baseline from the spectra, which are measured by a Fourier transform infrared spectrometer (FTIR), is an important preprocessing step for further spectra analysis such as quantitative and qualitative analysis. An automatic baseline correction method named iterative averaging, which is based on the basic knowledge of moving average, is presented. We also compared it to other methods, such as rubber band, adaptive iterative reweight penalized least squares, automatic iterative moving average, and morphological weighted penalized least squares, using simulated and experimental spectra with different signal-to-noise ratios (SNRs) to evaluate the performance of these methods by performance metrics and to select an appropriate method to analyze FTIR spectra. Performance metrics such as root-mean-square error, goodness-of-fit coefficient, and chi-square are calculated. The iterative averaging method achieves the best results, which are judged by performance metrics values, when it is applied to the FTIR spectra with different SNRs. It also can correct the baseline of the FTIR spectra automatically, and improve the capability and adaptability of the unsupervised online analysis of the FTIR system effectively.

Automatic baseline correction method for the open-path Fourier transform infrared spectra by using simple iterative averaging

It's necessary to remove the baseline from the spectra, which measured by open-path Fourier transform infrared spectrometry, for further spectral analysis such as qualitative and quantitative analysis. An automatic baseline correction method, the Iterative Averaging method, is presented. Baseline corrected by this method is accurate, and it also shows more precise than other methods when it is applied to Fourier Transform Infrared experimental spectra and simulated data. This method solves the key technology of the real-time on-line spectral analysis of OP-FTIR and improves the capability and adaptability of the unsupervised on-line system effectively.

An Automated Baseline Correction Method Based on Iterative Morphological Operations

Raman spectra usually suffer from baseline drift caused by fluorescence or other reasons. Therefore, baseline correction is a necessary and crucial step that must be performed before subsequent processing and analysis of Raman spectra. An automated baseline correction method based on iterative morphological operations is proposed in this work. The method can adaptively determine the structuring element first and then gradually remove the spectral peaks during iteration to get an estimated baseline. Experiments on simulated data and real-world Raman data show that the proposed method is accurate, fast, and flexible for handling different kinds of baselines in various practical situations. The comparison of the proposed method with some state-of-the-art baseline correction methods demonstrates its advantages over the existing methods in terms of accuracy, adaptability, and flexibility. Although only Raman spectra are investigated in this paper, the proposed method is hopefully to be used for the baseline correction of other analytical instrumental signals, such as IR spectra and chromatograms.

Salient space detection algorithm for signal extraction from contaminated and distorted spectrum

The proposed algorithm minimizes the influence of baseline distortion and exhibits good anti-noise ability and high real-time performance.

Background Subtraction of Raman Spectra Based on Iterative Polynomial Smoothing

In this paper, a novel background subtraction algorithm is presented that can automatically recover Raman signal. This algorithm is based on an iterative polynomial smoothing method that highly reduces the need for experience and a priori knowledge. First, a polynomial filter is applied to smooth the input spectrum (the input spectrum is just an original spectrum at the first iteration). The output curve of the filter divides the original spectrum into two parts, top and bottom. Second, a proportion is calculated between the lowest point of the signal in the bottom part and the highest point of the signal in the top part. The proportion is a key index that decides whether to go into a new iteration. If a new iteration is needed, the minimum value between the output curve and the original spectrum forms a new curve that goes into the same filter in the first step and continues as another iteration until no more iteration is needed to finally get the background of the original spectrum. Results from the simulation experiments not only show that the iterative polynomial smoothing algorithm achieves good performance, processing time, cost, and accuracy of recovery, but also prove that the algorithm adapts to different background types and a large signal-to-noise ratio range. Furthermore, real measured Raman spectra of organic mixtures and non-organic samples are used to demonstrate the application of the algorithm.

A concise iterative method using the Bezier technique for baseline construction

A novel approach, coined the Corner-Cutting method (CC, for short), is presented in this paper which affords the efficient construction of the baseline for analytical data streams. It was derived from techniques used in computer aided geometric design, a field established to produce curves and surfaces for the aviation and automobile industries. This corner-cutting technique provided a very efficient baseline calculation through an iterative process. Furthermore, a terminal condition was developed to make the process fully automated and truly non-parametric. Finally, we employed a Bezier curve to convert the iterating result into a smooth baseline solution. Compared to other iterative schemes used for baseline detection, our method was significantly efficient, easier to implement, and had a broader range of applications.

A batch correction method for liquid chromatography-mass spectrometry data that does not depend on quality control samples

The need for reproducible and comparable results is of increasing importance in non-targeted metabolomic studies, especially when differences between experimental groups are small. Liquid chromatography-mass spectrometry spectra are often acquired batch-wise so that necessary calibrations and cleaning of the instrument can take place. However this may introduce further sources of variation, such as differences in the conditions under which the acquisition of individual batches is performed. Quality control (QC) samples are frequently employed as a means of both judging and correcting this variation. Here we show that the use of QC samples can lead to problems. The non-linearity of the response can result in substantial differences between the recorded intensities of the QCs and experimental samples, making the required adjustment difficult to predict. Furthermore, changes in the response profile between one QC interspersion and the next cannot be accounted for and QC based correction can actually exacerbate the problems by introducing artificial differences. "Background correction" methods utilise all experimental samples to estimate the variation over time rather than relying on the QC samples alone. We compare non-QC correction methods with standard QC correction and demonstrate their success in reducing differences between replicate samples and their potential to highlight differences between experimental groups previously hidden by instrumental variation.

Hyperspectral image analysis. A tutorial

This tutorial aims at providing guidelines and practical tools to assist with the analysis of hyperspectral images. Topics like hyperspectral image acquisition, image pre-processing, multivariate exploratory analysis, hyperspectral image resolution, classification and final digital image processing will be exposed, and some guidelines given and discussed. Due to the broad character of current applications and the vast number of multivariate methods available, this paper has focused on an industrial chemical framework to explain, in a step-wise manner, how to develop a classification methodology to differentiate between several types of plastics by using Near infrared hyperspectral imaging and Partial Least Squares - Discriminant Analysis. Thus, the reader is guided through every single step and oriented in order to adapt those strategies to the user's case.

Ensemble learning for prediction of the bioactivity capacity of herbal medicines from chromatographic fingerprints

Background

Recent quality control of complex mixtures, including herbal medicines, is not limited to chemical chromatographic definition of one or two selected compounds; multivariate linear regression methods with dimension reduction or regularisation have been used to predict the bioactivity capacity from the chromatographic fingerprints of the herbal extracts. The challenge of this type of analysis requires a multi-dimensional approach at two levels: firstly each herb comprises complex mixtures of active and non-active chemical components; and secondly there are many factors relating to the growth, production, and processing of the herbal products. All these factors result in the significantly diverse concentrations of bioactive compounds in the herbal products. Therefore, it is imminent to have a predictive model with better generalisation that can accurately predict the bioactivity capacity of samples when only the chemical fingerprints data are available.

Results

In this study, the algorithm of Stacking Multivariate Linear Regression (SMLR) and a few other commonly used chemometric approaches were evaluated. They were to predict the Cluster of Differentiation 80 (CD80) expression bioactivity of a commonly used herb, Astragali Radix (AR), from the corresponding chemical chromatographic fingerprints. SMLR provides a superior prediction accuracy in comparison with the other multivariate linear regression methods of PCR, PLSR, OPLS and EN in terms of MSE_test and the goodness of prediction of test samples.

Conclusions

SMLR is a better platform than some multivariate linear regression methods. The first advantage of SMLR is that it has better generalisation to predict the bioactivity capacity of herbal medicines from their chromatographic fingerprints. Future studies should aim to further improve the SMLR algorithm. The second advantage of SMLR is that single chemical compounds can be effectively identified as highly bioactive components which demands further CD80 bioactivity confirmation..

Development and integration of block operations for data invariant automation of digital preprocessing and analysis of biological and biomedical Raman spectra

High-throughput information extraction from large numbers of Raman spectra is becoming an increasingly taxing problem due to the proliferation of new applications enabled using advances in instrumentation. Fortunately, in many of these applications, the entire process can be automated, yielding reproducibly good results with significant time and cost savings. Information extraction consists of two stages, preprocessing and analysis. We focus here on the preprocessing stage, which typically involves several steps, such as calibration, background subtraction, baseline flattening, artifact removal, smoothing, and so on, before the resulting spectra can be further analyzed. Because the results of some of these steps can affect the performance of subsequent ones, attention must be given to the sequencing of steps, the compatibility of these sequences, and the propensity of each step to generate spectral distortions. We outline here important considerations to effect full automation of Raman spectral preprocessing: what is considered full automation; putative general principles to effect full automation; the proper sequencing of processing and analysis steps; conflicts and circularities arising from sequencing; and the need for, and approaches to, preprocessing quality control. These considerations are discussed and illustrated with biological and biomedical examples reflecting both successful and faulty preprocessing.

Improved Savitzky-Golay-method-based fluorescence subtraction algorithm for rapid recovery of Raman spectra

In this paper, we propose an improved subtraction algorithm for rapid recovery of Raman spectra that can substantially reduce the computation time. This algorithm is based on an improved Savitzky-Golay (SG) iterative smoothing method, which involves two key novel approaches: (a) the use of the Gauss-Seidel method and (b) the introduction of a relaxation factor into the iterative procedure. By applying a novel successive relaxation (SG-SR) iterative method to the relaxation factor, additional improvement in the convergence speed over the standard Savitzky-Golay procedure is realized. The proposed improved algorithm (the RIA-SG-SR algorithm), which uses SG-SR-based iteration instead of Savitzky-Golay iteration, has been optimized and validated with a mathematically simulated Raman spectrum, as well as experimentally measured Raman spectra from non-biological and biological samples. The method results in a significant reduction in computing cost while yielding consistent rejection of fluorescence and noise for spectra with low signal-to-fluorescence ratios and varied baselines. In the simulation, RIA-SG-SR achieved 1 order of magnitude improvement in iteration number and 2 orders of magnitude improvement in computation time compared with the range-independent background-subtraction algorithm (RIA). Furthermore the computation time of the experimentally measured raw Raman spectrum processing from skin tissue decreased from 6.72 to 0.094 s. In general, the processing of the SG-SR method can be conducted within dozens of milliseconds, which can provide a real-time procedure in practical situations.

Automatic baseline recognition for the correction of large sets of spectra using continuous wavelet transform and iterative fitting

A new algorithm for the automatic recognition of peak and baseline regions in spectra is presented. It is part of a study to devise a baseline correction method that is particularly suitable for the simple and fast treatment of large amounts of data of the same type, such as those coming from high-throughput instruments, images, process monitoring, etc. This algorithm is based on the continuous wavelet transform, and its parameters are automatically determined using the criteria of Shannon entropy and the statistical distribution of noise, requiring virtually no user intervention. It was assessed on simulated spectra with different noise levels and baseline amplitudes, successfully recognizing the baseline points in all cases but for a few extremely weak and noisy signals. It can be combined with various fitting methods for baseline estimation and correction. In this work, it was used together with an iterative polynomial fitting to successfully process a real Raman image of 40,000 pixels in about 2.5 h.

Morphological weighted penalized least squares for background correction

Backgrounds existing in the analytical signal always impair the effectiveness of signals and compromise selectivity and sensitivity of analytical methods. In order to perform further qualitative or quantitative analysis, the background should be corrected with a reasonable method. For this purpose, a new automatic method for background correction, which is based on morphological operations and weighted penalized least squares (MPLS), has been developed in this paper. It requires neither prior knowledge about the background nor an iteration procedure or manual selection of a suitable local minimum value. The method has been successfully applied to simulated datasets as well as experimental datasets from different instruments. The results show that the method is quite flexible and could handle different kinds of backgrounds. The proposed MPLS method is implemented and available as an open source package at http://code.google.com/p/mpls.

FTIR spectroscopy of biofluids revisited: an automated approach to spectral biomarker identification

The extraction of disease specific information from Fourier transform infrared (FTIR) spectra of human body fluids demands the highest standards of accuracy and reproducibility of measurements because the expected spectral differences between healthy and diseased subjects are very small in relation to a large background absorbance of the whole sample. Here, we demonstrate that with the increased sensitivity of modern FTIR spectrometers, automatisation of sample preparation and modern bioinformatics, it is possible to identify and validate spectral biomarker candidates for distinguishing between urinary bladder cancer (UBC) and inflammation in suspected bladder cancer patients. The current dataset contains spectra of blood serum and plasma samples of 135 patients. All patients underwent cytology and pathological biopsy characterization to distinguish between patients without UBC (46) and confirmed UBC cases (89). A minimally invasive blood test could spare control patients a repeated cystoscopy including a transurethral biopsy, and three-day stationary hospitalisation. Blood serum, EDTA and citrate plasma were collected from each patient and processed following predefined strict standard operating procedures. Highly reproducible dry films were obtained by spotting sub-nanoliter biofluid droplets in defined patterns, which were compared and optimized. Particular attention was paid to the automatisation of sample preparation and spectral preprocessing to exclude errors by manual handling. Spectral biomarker candidates were identified from absorbance spectra and their 1(st) and 2(nd) derivative spectra using an advanced Random Forest (RF) approach. It turned out that the 2(nd) derivative spectra were most useful for classification. Repeat validation on 21% of the dataset not included in predictor training with Linear Discriminant Analysis (LDA) classifiers and Random Forests (RFs) yielded a sensitivity of 93 ± 10% and a specificity of 46 ± 18% for bladder cancer. The low specificity can be most likely attributed to the unbalanced and small number of control samples. Using this approach, spectral biomarker candidates in blood-derived biofluids were identified, which allow us to distinguish between cancer and inflammation, but the observed differences were tiny. Obviously, a much larger sample number has to be investigated to reliably validate such candidates.

An automated method for baseline correction, peak finding and peak grouping in chromatographic data

An automated method (FastChrom) for baseline correction, peak detection and assignment (grouping) of similar peaks across samples has been developed. The method has been tested both on artificial data and a dataset obtained from gas chromatograph analysis of wine samples. As part of the automated approach, a new method for baseline estimation has been developed and compared with other methods. FastChrom has been shown to perform at least as well as conventional software. However, compared to other approaches, FastChrom finds more peaks in the chromatograms and not only those with retention times defined by the user. FastChrom is fast and easy to use and offers the possibility of applying a retention time index which facilitates the identification of peaks and the comparison between experiments.

Imaging of plant cell walls by confocal Raman microscopy

Raman imaging of plant cell walls represents a nondestructive technique that can provide insights into chemical composition in context with structure at the micrometer level (<0.5 μm). The major steps of the experimental procedure are described: sample preparation (embedding and microcutting), setting the mapping parameters, and finally the calculation of chemical images on the basis of the acquired Raman spectra. Every Raman image is based on thousands of spectra, each being a spatially resolved molecular 'fingerprint' of the cell wall. Multiple components are analyzed within the native cell walls, and insights into polymer composition as well as the orientation of the cellulose microfibrils can be gained. The most labor-intensive step of this process is often the sample preparation, as the imaging approach requires a flat surface of the plant tissue with intact cell walls. After finishing the map (acquisition time is ∼10 min to 10 h, depending on the size of the region of interest and scanning parameters), many possibilities exist for the analysis of spectral data and image generation.

A small-window moving average-based fully automated baseline estimation method for Raman spectra

A fully automated and model-free baseline-correction method for vibrational spectra is presented. It iteratively applies a small, but increasing, moving average window in conjunction with peak stripping to estimate spectral baselines. Peak stripping causes the area stripped from the spectrum to initially increase and then diminish as peak stripping proceeds to completion; a subsequent increase is generally indicative of the commencement of baseline stripping. Consequently, this local minimum is used as a stopping criterion. A backup is provided by a second stopping criterion based on the area under a third-order polynomial fitted to the first derivative of the current estimate of the baseline-free spectrum and also indicates whether baseline is being stripped. When the second stopping criterion is triggered instead of the first one, a proportionally scaled simulated Gaussian baseline is added to the current estimate of the baseline-free spectrum to act as an internal standard to facilitate subsequent processing and termination via the first stopping criterion. The method is conceptually simple, easy to implement, and fully automated. Good and consistent results were obtained on simulated and real Raman spectra, making it suitable for the fully automated baseline correction of large numbers of spectra.