Automated spectral smoothing with spatially adaptive penalized least squares

Searching for Alternatives to the Savitzky-Golay Filter in the Spectral Processing Domain

An elegant, well-established effective data filter concept, proposed originally by Abraham Savitzky and Marcel J.E. Golay, is undoubtedly a very effective tool, however not free from limitations and drawbacks. Despite the latter, over the years it has become a "monopolist" in many fields of spectra processing, claiming a "commercial" superiority over alternative approaches, which would potentially allow to obtain equivalent or in some cases even more reliable results. In order to show that basic operations performed on spectral datasets, like smoothing or differentiation, do not have to be equated to the application of the one particular single algorithm, several of such alternatives are briefly presented within this paper and discussed with regard to their practical realization. A special emphasis is put on the fast Fourier methodology (FFT), being widespread in the general domain of signal processing. Finally, a user-friendly Matlab routine, in which the outlined algorithms are implemented, is shared, so that one can select and apply the technique of spectral data processing more adequate for their individual requirements without the need to code it prior to use.

Automated material identification with a Raman spectrometer based on the contribution enhancement of small differences and the adaptive target Raman peak subtraction

There is only a small difference in Raman peaks between two materials, but they also represent different molecular materials. Therefore, the accurate identification ability for similar materials with small differences among their Raman peaks plays a key role in Raman spectrometers for material identification. However, the noises, the baseline (i.e., fluorescence backgrounds), and the requirements, such as fast and automated detection, of excellent user experiences cause many difficulties. In this paper, the target Raman peak is directly subtracted from the detected Raman spectrum by the adaptive minimum root mean square error (RMSE) estimation for a residual spectrum. Unlike the usual methods in which the detected Raman peak needs to be first recovered by removing the baseline from its Raman spectrum and then to be compared with the target Raman peak, our method can effectively enhance the contribution of small differences between the detected and the target Raman peak on the residual spectrum so as to make the RMSE of the residual spectrum more sensitive with increasing differences. On the other hand, the obtained RMSE of the residual spectrum only has a small change for the detected Raman spectrum with various baselines. So the common criteria (i.e., the third-order polynomials describing RMSE) to identify the detected Raman spectrum with various baselines and the target Raman spectrum is presented. Simulation results show that the small difference, where there is only an additional small Raman peak as low as 1/25 of the maximum peak height, can also be accurately identified. Experiments also demonstrate that similar materials can be accurately identified, whereas some commercial Raman spectrometers fail to identify them. Our method effectively deals with the problem in which the error of the complex baseline correction causes erroneous judgement in Raman spectrometers for material identification.

Developing a Peak Extraction and Retention (PEER) Algorithm for Improving the Temporal Resolution of Raman Spectroscopy

In spectroscopic analysis, push-to-the-limit sensitivity is one of the important topics, particularly when facing the qualitative and quantitative analyses of the trace target. Normally, the effective recognition and extraction of weak signals are the first key steps, for which there has been considerable effort in developing various denoising algorithms for decades. Nevertheless, the lower the signal-to-noise ratio (SNR), the greater the deviation of the peak height and shape during the denoising process. Therefore, we propose a denoising algorithm along with peak extraction and retention (PEER). First, both the first and second derivatives of the Raman spectrum are used to determine Raman peaks with a high SNR whose peak information is kept away from the denoising process. Second, an optimized window smoothing algorithm is applied to the left part of the Raman spectrum, which is combined with the untreated Raman peaks to obtain the denoised Raman spectrum. The PEER algorithm is demonstrated with much better signal extraction and retention and successfully improves the temporal resolution of Raman imaging of a living cell by at least 1 order of magnitude higher than those by traditional algorithms.

Background Subtraction of Raman Spectra Based on Iterative Polynomial Smoothing

In this paper, a novel background subtraction algorithm is presented that can automatically recover Raman signal. This algorithm is based on an iterative polynomial smoothing method that highly reduces the need for experience and a priori knowledge. First, a polynomial filter is applied to smooth the input spectrum (the input spectrum is just an original spectrum at the first iteration). The output curve of the filter divides the original spectrum into two parts, top and bottom. Second, a proportion is calculated between the lowest point of the signal in the bottom part and the highest point of the signal in the top part. The proportion is a key index that decides whether to go into a new iteration. If a new iteration is needed, the minimum value between the output curve and the original spectrum forms a new curve that goes into the same filter in the first step and continues as another iteration until no more iteration is needed to finally get the background of the original spectrum. Results from the simulation experiments not only show that the iterative polynomial smoothing algorithm achieves good performance, processing time, cost, and accuracy of recovery, but also prove that the algorithm adapts to different background types and a large signal-to-noise ratio range. Furthermore, real measured Raman spectra of organic mixtures and non-organic samples are used to demonstrate the application of the algorithm.

Quantitative, Comparable Coherent Anti-Stokes Raman Scattering (CARS) Spectroscopy: Correcting Errors in Phase Retrieval

Coherent anti-Stokes Raman scattering (CARS) microspectroscopy has demonstrated significant potential for biological and materials imaging. To date, however, the primary mechanism of disseminating CARS spectroscopic information is through pseudocolor imagery, which explicitly neglects a vast majority of the hyperspectral data. Furthermore, current paradigms in CARS spectral processing do not lend themselves to quantitative sample-to-sample comparability. The primary limitation stems from the need to accurately measure the so-called nonresonant background (NRB) that is used to extract the chemically-sensitive Raman information from the raw spectra. Measurement of the NRB on a pixel-by-pixel basis is a nontrivial task; thus, reference NRB from glass or water are typically utilized, resulting in error between the actual and estimated amplitude and phase. In this manuscript, we present a new methodology for extracting the Raman spectral features that significantly suppresses these errors through phase detrending and scaling. Classic methods of error-correction, such as baseline detrending, are demonstrated to be inaccurate and to simply mask the underlying errors. The theoretical justification is presented by re-developing the theory of phase retrieval via the Kramers-Kronig relation, and we demonstrate that these results are also applicable to maximum entropy method-based phase retrieval. This new error-correction approach is experimentally applied to glycerol spectra and tissue images, demonstrating marked consistency between spectra obtained using different NRB estimates, and between spectra obtained on different instruments. Additionally, in order to facilitate implementation of these approaches, we have made many of the tools described herein available free for download.

Development and integration of block operations for data invariant automation of digital preprocessing and analysis of biological and biomedical Raman spectra

High-throughput information extraction from large numbers of Raman spectra is becoming an increasingly taxing problem due to the proliferation of new applications enabled using advances in instrumentation. Fortunately, in many of these applications, the entire process can be automated, yielding reproducibly good results with significant time and cost savings. Information extraction consists of two stages, preprocessing and analysis. We focus here on the preprocessing stage, which typically involves several steps, such as calibration, background subtraction, baseline flattening, artifact removal, smoothing, and so on, before the resulting spectra can be further analyzed. Because the results of some of these steps can affect the performance of subsequent ones, attention must be given to the sequencing of steps, the compatibility of these sequences, and the propensity of each step to generate spectral distortions. We outline here important considerations to effect full automation of Raman spectral preprocessing: what is considered full automation; putative general principles to effect full automation; the proper sequencing of processing and analysis steps; conflicts and circularities arising from sequencing; and the need for, and approaches to, preprocessing quality control. These considerations are discussed and illustrated with biological and biomedical examples reflecting both successful and faulty preprocessing.

Adaptive total variation-based spectral deconvolution with the split Bregman method

Spectroscopic data often suffer from common problems of band overlap and noise. This paper presents a maximum a posteriori (MAP)-based algorithm for the band overlap problem. In the MAP framework, the likelihood probability density function (PDF) is constructed with Gaussian noise assumed, and the prior PDF is constructed with adaptive total variation (ATV) regularization. The split Bregman iteration algorithm is employed to optimize the ATV spectral deconvolution model and accelerate the speed of the spectral deconvolution. The main advantage of this algorithm is that it can obtain peak structure information as well as suppress noise simultaneity. Simulated and real spectra experiments manifest that this algorithm can satisfactorily recover the overlap peaks as well as suppress noise and are robust to the regularization parameter.

A MAP-based algorithm for spectroscopic semi-blind deconvolution

Spectroscopic data often suffer from common problems of bands overlapping and random noise. In this paper, we show that the issue of overlapping peaks can be considered as a maximum a posterior (MAP) problem and be solved by minimizing an object functional that includes a likelihood term and two prior terms. In the MAP framework, the likelihood probability density function (PDF) is constructed based on a spectral observation model, a robust Huber-Markov model is used as spectra prior PDF, and the kernel prior is described based on a parametric Gaussian function. Moreover, we describe an efficient optimization scheme that alternates between latent spectrum recovery and blur kernel estimation until convergence. The major novelty of the proposed algorithm is that it can estimate the kernel slit width and latent spectrum simultaneously. Comparative results with other deconvolution methods suggest that the proposed method can recover spectral structural details as well as suppress noise effectively.