PCA and PLS Analysis of Lanthanides Using Absorbance and Single-Beam Visible Spectra

During process monitoring applications, referenced optical spectroscopy, such as absorbance spectroscopy, can suffer from environmental and instrumental fluctuations that alter the intensity of irradiance reaching the spectrometer's detector at each detected frequency. Temperature, vibration, light source aging, instrument damage, detector aging, detector registry shifts, sampling cell degradation, and similar perturbations create situations in which a previously collected reference spectrum may no longer be valid for the current state of the system. This can lead to the calculation of poor-quality absorbance spectra that are unsuitable for qualitative or quantitative analysis based on prior calibration models. The use of single-beam spectra in the creation of multivariate calibration models circumvents the need for collecting and maintaining a stable reference spectrum throughout an ongoing chemical process. However, unlike absorbance spectra, which typically have a zero baseline, single-beam spectra contain a high background signal relative to an analyte signal, and they may also contain intense peaks from the light source. Here, multivariate principal component analysis (PCA) and partial least squares (PLS) regression models are built using single-beam and absorbance spectra to compare the efficacy of both types of spectra for qualitative and quantitative analyses of lanthanide solutions. A multileg fiber optic UV-visible spectrometer is utilized to collect samples under three distinct wavelength registries in three unique sampling cells and under lighting conditions spanning 0.2 to 2.0 relative transmittance. Under these conditions, single-beam spectral PCA models produced enhanced discrimination between sampling conditions, allowing spectra to be grouped by the instrumental conditions under which they were collected. Absorbance and single-beam PLS models produced equivalent quantitations of the lanthanide concentrations.

NIR quantitative model trans-scale calibration from small scale to pilot scale via directed DOSC-SBC algorithm

In order to solve the problem of inapplicability of NIR quantitative models due to the large difference between the modeling samples and the samples to be tested, Directed DOSC-SBC（DDOSC-SBC）algorithm is proposed in this paper based on Direct Orthogonal Signal Correction combined with Slope/Bias Correction (DOSC-SBC) algorithm. To obtain the suitable spectral matrix transfer parameters for the test set during DDOSC spectral preprocessing, several representative test samples in the test set were selected, then the spectral systematic errors between the modeling set and the test set were corrected with the SBC method in order to realize the trans-scale prediction of the NIR quantitative model. NIR data and the critical quality attributes（CQAs）were detected in the small scale and pilot scale pharmaceutical process of the fluidized bed granulation of dextrin and water extraction of honeysuckle. After the small scale model was calibrated via the directed DOSC-SBC algorithm which was guided by representative pilot scale samples, the small scale model was able to predict the pilot scale test samples more accurately. The NIR quantitative model trans-scale calibration from small scale to pilot scale was also successfully realized with a RPD value higher than 3.5 and RSEP value lower than 10%. DDOSC-SBC algorithm is a successful model trans-scale calibrated method that can be applied to NIR real-time monitoring of CQAs in the preparation process of Chinese herbal medicine.

Application of adaptive Laplacian Eigenmaps in near infrared spectral modeling

Laplacian Eigenmaps is a nonlinear dimensionality reduction algorithm based on graph theory. The algorithm adopted the Gaussian function to measure the affinity between a pair of points in the adjacency graph. However, the scaling parameter σ in the Gaussian function is a hyper-parameter tuned empirically. Once the value of σ is determined and fixed, the weight between two points depends wholly on the Euclidian distance between them, which is not suitable for multi-scale sample sets. To optimize the weight between two points in the adjacency graph and make the weight reflect the scale information of different sample sets, an adaptive LE improved algorithm is used in this paper. Considering the influence of adjacent sample points and multi-scale data, the Euclidean distance between the k-th nearest sample point to sample point x_i is used as the local scaling parameter σ_i of x_i, instead of using a single scaling parameter σ. The efficiency of the algorithm is testified by applying on two public near-infrared data sets. LE-SVR and ALE-SVR models are established after LE and ALE dimension reduction of SNV preprocessed data sets. Compared with the LE-SVR model, the R² and RPD of the ALE-SVR model established on the two data sets are improved, while RMSE is decreased, indicating that the prediction effect and stability of the regression model are established by the ALE algorithm are better than that of the traditional LE algorithm. Experiments show that the ALE algorithm can achieve a better dimensionality reduction effect than the LE algorithm.

Spectral Encoder to Extract the Features of Near-Infrared Spectra for Multivariate Calibration

An autoencoder architecture was adopted for near-infrared (NIR) spectral analysis by extracting the common features in the spectra. Three autoencoder-based networks with different purposes were constructed. First, a spectral encoder was established by training the network with a set of spectra as the input. The features of the spectra can be encoded by the nodes in the bottleneck layer, which in turn can be used to build a sparse and robust model. Second, taking the spectra of one instrument as the input and that of another instrument as the reference output, the common features in both spectra can be obtained in the bottleneck layer. Therefore, in the prediction step, the spectral features of the second can be predicted by taking the reverse of the decoder as the encoder. Furthermore, transfer learning was used to build the model for the spectra of more instruments by fine-tuning the trained network. NIR datasets of plant, wheat, and pharmaceutical tablets measured on multiple instruments were used to test the method. The multi-linear regression (MLR) model with the encoded features was found to have a similar or slightly better performance in prediction compared with the partial least-squares (PLS) model.

Calibration model transfer in mid-infrared process analysis with in situ attenuated total reflectance immersion probes

Process applications of mid-infrared (MIR) spectrometry may involve replacement of the spectrometer and/or measurement probe, which generally requires a calibration transfer method to maintain the accuracy of analysis. In this study, direct standardisation (DS), piecewise direct standardisation (PDS) and spectral space transformation (SST) were compared for analysis of ternary mixtures of acetone, ethanol and ethyl acetate. Three calibration transfer examples were considered: changing the spectrometer, multiplexing two probes to a spectrometer, and changing the diameter of the attenuated total reflectance (ATR) probe (as might be required when scaling up from lab to process analysis). In each case, DS, PDS and SST improved the accuracy of prediction for the test samples, analysed on a secondary spectrometer-probe combination, using a calibration model developed on the primary system. When the probe diameter was changed, a scaling step was incorporated into SST to compensate for the change in absorbance caused by the difference in ATR crystal size. SST had some advantages over DS and PDS: DS was sensitive to the choice of standardisation samples, and PDS required optimisation of the window size parameter (which also required an extra standardisation sample). SST only required a single parameter to be chosen: the number of principal components, which can be set equal to the number of standardisation samples when a low number of standards (n < 7) are used, which is preferred to minimise the time required to transfer the calibration model.

Comparison of Spectral Reflectance-Based Smart Farming Tools and a Conventional Approach to Determine Herbage Mass and Grass Quality on Farm

The analysis of multispectral imagery (MSI) acquired by unmanned aerial vehicles (UAVs) and mobile near-infrared reflectance spectroscopy (NIRS) used on-site has become increasingly promising for timely assessments of grassland to support farm management. However, a major challenge of these methods is their calibration, given the large spatiotemporal variability of grassland. This study evaluated the performance of two smart farming tools in determining fresh herbage mass and grass quality (dry matter, crude protein, and structural carbohydrates): an analysis model for MSI (GrassQ) and a portable on-site NIRS (HarvestLabTM 3000). We compared them to conventional look-up tables used by farmers. Surveys were undertaken on 18 multi-species grasslands located on six farms in Switzerland throughout the vegetation period in 2018. The sampled plots represented two phenological growth stages, corresponding to an age of two weeks and four to six weeks, respectively. We found that neither the performance of the smart farming tools nor the performance of the conventional approach were satisfactory for use on multi-species grasslands. The MSI-model performed poorly, with relative errors of 99.7% and 33.2% of the laboratory analyses for herbage mass and crude protein, respectively. The errors of the MSI-model were indicated to be mainly caused by grassland and environmental characteristics that differ from the relatively narrow Irish calibration dataset. The On-site NIRS showed comparable performance to the conventional Look-up Tables in determining crude protein and structural carbohydrates (error ≤ 22.2%). However, we identified that the On-site NIRS determined undried herbage quality with a systematic and correctable error. After corrections, its performance was better than the conventional approach, indicating a great potential of the On-site NIRS for decision support on grazing and harvest scheduling.

Three-level simultaneous component analysis for analyzing the near-infrared spectra of aqueous solutions under multiple perturbations

Quantitative analysis under various perturbations is a difficult problem because the analytical signal changes with different factors. In this work, three-level simultaneous component analysis (3-MSCA) was used for analyzing the near-infrared (NIR) spectra of aqueous solutions under different perturbations. The spectral data of aqueous proline solutions at different pH, concentration and temperature were measured, and a three-level model was built to describe the effects of the three perturbations on the spectra, respectively. The first level model describes the change of the spectra with pH, from which significant aggregation of proline was observed around the isoelectric point. The second and third level model show the spectral change with concentration and temperature, respectively, and the spectral feature has a very good linear relationship with the corresponding influencing factors. Therefore, the pH and concentration scores can be used as the calibration curve for quantitative analysis of the pH and the content of proline, and the temperature scores can be used to predict the temperature of the solutions. In addition, the structural change of water molecules under different conditions is obtained from the loadings. A decline of the bulk water was found with the increase of concentration, implying an ascending trend of the bonded water due to the interaction of proline and water. The dissociation of water clusters with the increase of temperature is also displayed.

Stacked ensemble extreme learning machine coupled with Partial Least Squares-based weighting strategy for nonlinear multivariate calibration

With its simple theory and strong implementation, extreme learning machine (ELM) becomes a competitive single hidden layer feed forward networks for nonlinear multivariate calibration in chemometrics. To improve the generalization and robustness of ELM further, stacked generalization is introduced into ELM to construct a modified ELM model called stacked ensemble ELM (SE-ELM). The SE-ELM is to create a set of sub-models by applying ELM repeatedly to different sub-regions of the spectra and then combine the predictions of those sub-models according to a weighting strategy. Three different weighting strategies are explored to implement the proposed SE-ELM, such as the Winner-takes-all (WTA) weighting strategy, the constraint non-negative least squares (CNNLS) weighing strategy and the partial least squares (PLS) weighting strategy. Furthermore, PLS is suggested to be selected as the optimal weighting method that can handle the multi-colinearity among the predictions yielded by all the sub-models. The experimental assessment of the three SE-ELM models with different weighting strategies is carried out on six real spectroscopic datasets and compared with ELM, back-propagation neural network (BPNN) and Radial basis function neural network (RBFNN), statistically tested by the Wilcoxon signed rank test. The obtained experimental results suggest that, in general, all the SE-ELM models are more robust and more accurate than traditional ELM. In particular, the proposed PLS-based weighting strategy is at least statistically not worse than, and frequently better than the other two weighting strategies, BPNN, and RBFNN.

Calibration Transfer Based on Affine Invariance for NIR without Transfer Standards

Calibration transfer is an important field for near-infrared (NIR) spectroscopy in practical applications. However, most transfer methods are constructed with standard samples, which are expensive and difficult to obtain. Taking this problem into account, this paper proposes a calibration transfer method based on affine invariance without transfer standards (CTAI). Our method can be utilized to adjust the difference between two instruments by affine transformation. CTAI firstly establishes a partial least squares (PLS) model of the master instrument to obtain score matrices and predicted values of the two instruments, and then the regression coefficients between each of the score vectors and predicted values are computed for the master instrument and the slave instrument, respectively. Next, angles and biases are calculated between the regression coefficients of the master instrument and the corresponding regression coefficients of the slave instrument, respectively. Finally, by introducing affine transformation, new samples are predicted based on the obtained angles and biases. A comparative study between CTAI and the other five methods was conducted, and the performances of these algorithms were tested with two NIR spectral datasets. The obtained experimental results show clearly that, in general CTAI is more robust and can also achieve the best Root Mean Square Error of test sets (RMSEPs). In addition, the results of statistical difference with the Wilcoxon signed rank test show that CTAI is generally better than the others, and at least statistically the same.

Multi-spectrometer calibration transfer based on independent component analysis

Calibration transfer is indispensable for practical applications of near infrared (NIR) spectroscopy due to the need for precise and consistent measurements across different spectrometers. In this work, a method for multi-spectrometer calibration transfer is described based on independent component analysis (ICA). A spectral matrix is first obtained by aligning the spectra measured on different spectrometers. Then, by using independent component analysis, the aligned spectral matrix is decomposed into the mixing matrix and the independent components of different spectrometers. These differing measurements between spectrometers can then be standardized by correcting the coefficients within the independent components. Two NIR datasets of corn and edible oil samples measured with three and four spectrometers, respectively, were used to test the reliability of this method. The results of both datasets reveal that spectra measurements across different spectrometers can be transferred simultaneously and that the partial least squares (PLS) models built with the measurements on one spectrometer can predict that the spectra can be transferred correctly on another.

Linear model correction: A method for transferring a near-infrared multivariate calibration model without standard samples

Calibration transfer is essential for practical applications of near infrared (NIR) spectroscopy because the measurements of the spectra may be performed on different instruments and the difference between the instruments must be corrected. For most of calibration transfer methods, standard samples are necessary to construct the transfer model using the spectra of the samples measured on two instruments, named as master and slave instrument, respectively. In this work, a method named as linear model correction (LMC) is proposed for calibration transfer without standard samples. The method is based on the fact that, for the samples with similar physical and chemical properties, the spectra measured on different instruments are linearly correlated. The fact makes the coefficients of the linear models constructed by the spectra measured on different instruments are similar in profile. Therefore, by using the constrained optimization method, the coefficients of the master model can be transferred into that of the slave model with a few spectra measured on slave instrument. Two NIR datasets of corn and plant leaf samples measured with different instruments are used to test the performance of the method. The results show that, for both the datasets, the spectra can be correctly predicted using the transferred partial least squares (PLS) models. Because standard samples are not necessary in the method, it may be more useful in practical uses.

Quantitative determination based on the differences between spectra-temperature relationships

In the Near-infrared (NIR) spectral measurement it is not always possible to keep the experimental conditions constant. The fluctuations in external variables, such as temperature, will result in a nonlinear shift and a broadening of the spectral bands. In this study, the temperature-induced spectral variation coefficient (TSVC) was obtained by using loading space standardization (LSS). The relationship between TSVC and normalized squared temperature was quantitatively analyzed and applied to the quantitative determination of the compositions in mixtures. NIR spectra of peanut-soy-corn oil mixtures measured at seven temperatures were analyzed. It was found that, the relationship between TSVC and normalized squared temperature can be established by using LSS. Furthermore, the quantitative determination of the compositions in a mixture can be achieved by using the difference between the relationships, i.e., the slope of the relationship. The calibration curves between slope and composition volume are found to be reliable with the correlation coefficients (R(2)) as high as 0.9992. Quantitative determination by the calibration curves were also validated. Therefore, the method can be an effective tool for investigating the effect of temperature and quantitatively analysis.

Calibration Transfer Between a Bench Scanning and a Submersible Diode Array Spectrophotometer for In Situ Wastewater Quality Monitoring in Sewer Systems

Online monitoring programs based on spectroscopy have a high application potential for the detection of hazardous wastewater discharges in sewer systems. Wastewater hydraulics poses a challenge for in situ spectroscopy, especially when the system includes storm water connections leading to rapid changes in water depth, velocity, and in the water quality matrix. Thus, there is a need to optimize and fix the location of in situ instruments, limiting their availability for calibration. In this context, the development of calibration models on bench spectrophotometers to estimate wastewater quality parameters from spectra acquired with in situ instruments could be very useful. However, spectra contain information not only from the samples, but also from the spectrophotometer generally invalidating this approach. The use of calibration transfer methods is a promising solution to this problem. In this study, calibration models were developed using interval partial least squares (iPLS), for the estimation of total suspended solids (TSS) and chemical oxygen demand (COD) in sewage from Ultraviolet-visible spectra acquired in a bench scanning spectrophotometer. The feasibility of calibration transfer to a submersible, diode array equipment, to be subsequently operated in situ, was assessed using three procedures: slope and bias correction (SBC); single wavelength standardization (SWS) on mean spectra; and local centering (LC). The results showed that SBC was the most adequate for the available data, adding insignificant error to the base model estimates. Single wavelength standardization was a close second best, potentially more robust, and independent of the base iPLS model. Local centering was shown to be inadequate for the samples and instruments used.