Maintaining the predictive abilities of multivariate calibration models by spectral space transformation

Supervised Factor Analysis Transfer: Calibration transfer with noise modeling and response variable integration

Multivariate calibration models often encounter challenges in extrapolating beyond the calibration instruments due to variations in hardware configurations, signal processing algorithms, or environmental conditions. Calibration transfer techniques have been developed to mitigate this issue. In this study, we introduce a novel methodology known as Supervised Factor Analysis Transfer (SFAT) aimed at achieving robust and interpretable calibration transfer. SFAT operates from a probabilistic framework and integrates response variables into its transfer process to effectively align data from the target instrument to that of the source instrument. Within the SFAT model, the data from the source instrument, the target instrument, and the response variables are collectively projected onto a shared set of latent variables. These latent variables serve as the conduit for information transfer between the three distinct domains, thereby facilitating effective spectra transfer. Moreover, SFAT explicitly models the noise variances associated with each variable, thereby minimizing the transfer of non-informative noise. Furthermore, we provide empirical evidence showcasing the efficacy of SFAT across three real-world datasets, demonstrating its superior performance in calibration transfer scenarios.

Calibration transfer between NIR instruments using optimally predictive calibration subsets

In this study, a new approach for the selection of informative standardization samples from the original calibration set for the transfer of a calibration model between NIR instruments is proposed and evaluated. First, a calibration model is developed, after variable selection by the Final Complexity Adapted Models (FCAM) method, using the significance of the PLS regression coefficients (FCAM-SIG) as selection criterion. Then, the resulting model is used for the selection of the best fitting subset of calibration samples with optimally predictive ability, called the optimally predictive calibration subset (OPCS). Next, the standardization samples are selected from the OPCS. The spectra on the slave instruments are transferred to corresponding spectra on the master instrument by the widely used Piecewise Direct Standardization (PDS) method. Thereafter, for the test set on the slave instrument, a 3D response surface plot is drawn for the root mean squared error of prediction (RMSEP) as a function of the number of OPCS samples and window sizes used for the PDS method. Finally, the smallest set of calibration samples, in combination with the optimal window size, providing the optimal RMSEP, is selected as standardization set. The proposed OPCS approach for the selection of standardization samples is tested on two real-life NIR data sets providing 13 X-y combinations to model. The results show that the obtained numbers of OPCS-based standardization samples are statistically significantly lower than those obtained with the widely used representative sample selection method of Kennard and Stone, while the predictive performances are similar.

Rapid Classification and Differentiation of Sepsis-Related Pathogens Using FT-IR Spectroscopy

Sepsis is a life-threatening condition arising from a dysregulated host immune response to infection, leading to a substantial global health burden. The accurate identification of bacterial pathogens in sepsis is essential for guiding effective antimicrobial therapy and optimising patient outcomes. Traditional culture-based bacterial typing methods present inherent limitations, necessitating the exploration of alternative diagnostic approaches. This study reports the successful application of Fourier-transform infrared (FT-IR) spectroscopy in combination with chemometrics as a potent tool for the classification and discrimination of microbial species and strains, primarily sourced from individuals with invasive infections. These samples were obtained from various children with suspected sepsis infections with bacteria and fungi originating at different sites. We conducted a comprehensive analysis utilising 212 isolates from 14 distinct genera, comprising 202 bacterial and 10 fungal isolates. With the spectral analysis taking several weeks, we present the incorporation of quality control samples to mitigate potential variations that may arise between different sample plates, especially when dealing with a large sample size. The results demonstrated a remarkable consistency in clustering patterns among 14 genera when subjected to principal component analysis (PCA). Particularly, Candida, a fungal genus, was distinctly recovered away from bacterial samples. Principal component discriminant function analysis (PC-DFA) allowed for distinct discrimination between different bacterial groups, particularly Gram-negative and Gram-positive bacteria. Clear differentiation was also observed between coagulase-negative staphylococci (CNS) and Staphylococcus aureus isolates, while methicillin-resistant S. aureus (MRSA) was also separated from methicillin-susceptible S. aureus (MSSA) isolates. Furthermore, highly accurate discrimination was achieved between Enterococcus and vancomycin-resistant enterococci isolates with 98.4% accuracy using partial least squares-discriminant analysis. The study also demonstrates the specificity of FT-IR, as it effectively discriminates between individual isolates of Streptococcus and Candida at their respective species levels. The findings of this study establish a strong groundwork for the broader implementation of FT-IR and chemometrics in clinical and microbiological applications. The potential of these techniques for enhanced microbial classification holds significant promise in the diagnosis and management of invasive bacterial infections, thereby contributing to improved patient outcomes.

Calibration set reduction by the selection of a subset containing the best fitting samples showing optimally predictive ability

Near-infrared (NIR) spectroscopy is a rapid, non-invasive and cost-effective technique, for which sample pre-treatment is often not required. It is applied for both qualitative and quantitative analyses in various application fields. Often, large calibration sets are used, from which informative subsets can be selected without a loss of meaningful information. In this study, a new approach for sample subset selection is proposed and evaluated. The global PLS model, obtained with the original large global calibration set after FCAM-SIG variable selection, is used for the selection of the best fitting subset of calibration samples with optimally predictive ability. This best fitting calibration subset is called the optimally predictive calibration subset (OPCS). After ranking the global calibration samples according to increasing residuals, different enlarging fractions of the ranked calibration set are selected. For each fraction, the optimal predictive ability and the corresponding optimal PLS complexity are determined by cross model validation (CMV). After performing CMV with all fractions, the fraction with the best fitting samples and optimally predictive ability, i.e. the OPCS, is determined. The use of the best fitting samples from the global PLS model results in an OPCS-based model which is similar to the global PLS model and has a similar predictive ability. Because the best fitting samples do not need to be representative for the global calibration set, but only need to support the OPCS-based model, the number of samples in the OPCS model is mostly smaller than that selected by a traditional representative sample subset selection method. The new OPCS approach is tested on three real life NIR data sets with twelve X-y combinations to model. The results show that the number of selected samples obtained by the OPCS approach is statistically significantly lower than (i) that of the most suitable and widely used representative sample selection method of Kennard and Stone, and (ii) that suggested by the guideline that the optimal sample size N for reduced calibration sets should surpass the PLS model complexity A by a factor 12. An additional advantage of the OPCS approach is that no outliers are included in the subset because only the best fitting calibration samples are selected. In the new OPCS approach, two additional innovations are built in: (i) CMV is for the first time applied for sample selection and (ii) in CMV, the "one standard error rule", adopted from "Repeated Double Cross Validation", is for the first time used for the determination of the optimal PLS complexity of the OPCS-based models.

PFCE2: A versatile parameter-free calibration enhancement framework for near-infrared spectroscopy

Near-infrared (NIR) spectroscopy is a widely used technique for chemical analysis, but it has faced challenges of calibration transfer, maintenance, and enhancement among different instruments and conditions. The parameter-free calibration enhancement (PFCE) framework was developed to address these challenges with non-supervised (NS), semi-supervised (SS), and full-supervised (FS) methods. This study presented PFCE2, an updated version of the PFCE framework that incorporates two new constraints and a new method to improve the robustness and efficiency of calibration enhancement. First, normalized L2 and L1 constraints were introduced to replace the correlation coefficient (Corr) constraint used in the original PFCE. These constraints preserve the parameter-free feature of PFCE and impose smoothness or sparsity on the model coefficients. Second, multitask PFCE (MT-PFCE) was proposed within the framework to address the calibration enhancement among multiple instruments, enabling the framework to be versatile for all possible calibration transfer situations. Demonstrations conducted on three NIR datasets of tablets, plant leaves, and corn showed that the PFCE methods with the new L2 and L1 constraints can result in more accurate and robust predictions than the Corr constraint, especially when the standard sample size is small. Moreover, MT-PFCE could refine all models in the involved scenarios at once, leading to significant enhancement in model performance, compared to the original PFCE method with the same data requirements. Finally, the applicable situations of the PFCE framework and other analogous calibration transfer methods were summarized, facilitating users to choose suitable methods for their application. The source codes written in both MATLAB and Python are available at https://github.com/JinZhangLab/PFCE and https://pypi.org/project/pynir/, respectively.

Improved Principal Component Analysis (IPCA): A Novel Method for Quantitative Calibration Transfer between Different Near-Infrared Spectrometers

Given the labor-consuming nature of model establishment, model transfer has become a considerable topic in the study of near-infrared (NIR) spectroscopy. Recently, many new algorithms have been proposed for the model transfer of spectra collected by the same types of instruments under different situations. However, in a practical scenario, we need to deal with model transfer between different types of instruments. To expand model applicability, we must develop a method that could transfer spectra acquired from different types of NIR spectrometers with different wavenumbers or absorbance. Therefore, in our study, we propose a new methodology based on improved principal component analysis (IPCA) for calibration transfer between different types of spectrometers. We adopted three datasets for method evaluation, including public pharmaceutical tablets (dataset 1), corn data (dataset 2), and the spectra of eight batches of samples acquired from the plasma ethanol precipitation process collected by FT-NIR and MicroNIR spectrometers (dataset 3). In the calibration transfer for public datasets, IPCA displayed comparable results with the classical calibration transfer method using piecewise direct standardization (PDS), indicating its obvious ability to transfer spectra collected from the same types of instruments. However, in the calibration transfer for dataset 3, our proposed IPCA method achieved a successful bi-transfer between the spectra acquired from the benchtop and micro-instruments with/without wavelength region selection. Furthermore, our proposed method enabled improvements in prediction ability rather than the degradation of the models built with original micro spectra. Therefore, our proposed method has no limitations on the spectrum for model transfer between different types of NIR instruments, thus allowing a wide application range, which could provide a supporting technology for the practical application of NIR spectroscopy.

Prediction approach of larch wood density from visible-near-infrared spectroscopy based on parameter calibrating and transfer learning

Wood density, as a key indicator to measure wood properties, is of weighty significance in enhancing wood utilization and modifying wood properties in sustainable forest management. Visible-near-infrared (Vis-NIR) spectroscopy provides a feasible and efficient solution for obtaining wood density by the advantages of its efficiency and non-destructiveness. However, the spectral responses are different in wood products with different moisture content conditions, and changes in external factors may cause the regression model to fail. Although some calibration transfer methods and convolutional neural network (CNN)-based deep transfer learning methods have been proposed, the generalization ability and prediction accuracy of the models still need to be improved. For the prediction problem of Vis-NIR wood density in different moisture contents, a deep transfer learning hybrid method with automatic calibration capability (Resnet1D-SVR-TrAdaBoost.R2) was proposed in this study. The disadvantage of overfitting was avoided when CNN processes small sample data, which considered the complex exterior factors in actual production to enhance feature extraction and migration between samples. Density prediction of the method was performed on a larch dataset with different moisture content conditions, and the hybrid method was found to achieve the best prediction results under the calibration samples with different target domain calibration samples and moisture contents, and the performance of models was better than that of the traditional calibration transfer and migration learning methods. In particular, the hybrid model has achieved an improvement of about 0.1 in both R ² and root mean square error (RMSE) values compared to the support vector regression model transferred by piecewise direct standardization method (SVR+PDS), which has the best performance among traditional calibration methods. To further ascertain the generalizability of the hybrid model, the model was validated with samples collected from mixed moisture contents as the target domain. Various experiments demonstrated that the Resnet1D-SVR-TrAdaBoost.R2 model could predict larch wood density with a high generalization ability and accuracy effectively but was computation consuming. It showed the potential to be extended to predict other metrics of wood.

Water as a Probe for Standardization of Near-Infrared Spectra by Mutual-Individual Factor Analysis

The standardization of near-infrared (NIR) spectra is essential in practical applications, because various instruments are generally employed. However, standardization is challenging due to numerous perturbations, such as the instruments, testing environments, and sample compositions. In order to explain the spectral changes caused by the various perturbations, a two-step standardization technique was presented in this work called mutual–individual factor analysis (MIFA). Taking advantage of the sensitivity of a water probe to perturbations, the spectral information from a water spectral region was gradually divided into mutual and individual parts. With aquaphotomics expertise, it can be found that the mutual part described the overall spectral features among instruments, whereas the individual part depicted the difference of component structural changes in the sample caused by operation and the measurement conditions. Furthermore, the spectral difference was adjusted by the coefficients in both parts. The effectiveness of the method was assessed by using two NIR datasets of corn and wheat, respectively. The results showed that the standardized spectra can be successfully predicted by using the partial least squares (PLS) models developed with the spectra from the reference instrument. Consequently, the MIFA offers a viable solution to standardize the spectra obtained from several instruments when measurements are affected by multiple factors.

Unsupervised dynamic orthogonal projection. An efficient approach to calibration transfer without standard samples

Calibration transfer has been traditionally performed in the context of transferring models between instruments using standard samples. Recently, new methodologies and applications have shown that transfer techniques can be adopted to achieve calibration transfer between other types of domains, such as product form, variant or seasonality. In addition, to achieving a higher efficiency for calibration transfer, it is desirable to perform the transfer without the need for standard samples or new reference analyses. Therefore, we propose a method for unsupervised calibration transfer based on the orthogonalization for structural differences between domains. The method has been successfully applied to one simulated dataset and two real datasets. In the studied cases, the proposed methodology allowed to achieve a successful transfer of calibration models and enabled the interpretation of the interferences responsible for the degradation of the original calibration models when transferred to the new domain.

Spectral Encoder to Extract the Features of Near-Infrared Spectra for Multivariate Calibration

An autoencoder architecture was adopted for near-infrared (NIR) spectral analysis by extracting the common features in the spectra. Three autoencoder-based networks with different purposes were constructed. First, a spectral encoder was established by training the network with a set of spectra as the input. The features of the spectra can be encoded by the nodes in the bottleneck layer, which in turn can be used to build a sparse and robust model. Second, taking the spectra of one instrument as the input and that of another instrument as the reference output, the common features in both spectra can be obtained in the bottleneck layer. Therefore, in the prediction step, the spectral features of the second can be predicted by taking the reverse of the decoder as the encoder. Furthermore, transfer learning was used to build the model for the spectra of more instruments by fine-tuning the trained network. NIR datasets of plant, wheat, and pharmaceutical tablets measured on multiple instruments were used to test the method. The multi-linear regression (MLR) model with the encoded features was found to have a similar or slightly better performance in prediction compared with the partial least-squares (PLS) model.

Calibration model transfer in mid-infrared process analysis with in situ attenuated total reflectance immersion probes

Process applications of mid-infrared (MIR) spectrometry may involve replacement of the spectrometer and/or measurement probe, which generally requires a calibration transfer method to maintain the accuracy of analysis. In this study, direct standardisation (DS), piecewise direct standardisation (PDS) and spectral space transformation (SST) were compared for analysis of ternary mixtures of acetone, ethanol and ethyl acetate. Three calibration transfer examples were considered: changing the spectrometer, multiplexing two probes to a spectrometer, and changing the diameter of the attenuated total reflectance (ATR) probe (as might be required when scaling up from lab to process analysis). In each case, DS, PDS and SST improved the accuracy of prediction for the test samples, analysed on a secondary spectrometer-probe combination, using a calibration model developed on the primary system. When the probe diameter was changed, a scaling step was incorporated into SST to compensate for the change in absorbance caused by the difference in ATR crystal size. SST had some advantages over DS and PDS: DS was sensitive to the choice of standardisation samples, and PDS required optimisation of the window size parameter (which also required an extra standardisation sample). SST only required a single parameter to be chosen: the number of principal components, which can be set equal to the number of standardisation samples when a low number of standards (n < 7) are used, which is preferred to minimise the time required to transfer the calibration model.

Algorithm of Stability-Analysis-Based Feature Selection for NIR Calibration Transfer

For conventional near-infrared spectroscopy (NIR) technology, even within the same sample, the NIR spectral signal can vary significantly with variation of spectrometers and the spectral collection environment. In order to improve the applicability and application of NIR prediction models, effective calibration transfer is essential. In this study, a stability-analysis-based feature selection algorithm (SAFS) for NIR calibration transfer is proposed, which is used to extract effective spectral band information with high stability between the master and slave instruments during the calibration transfer process. The stability of the spectrum bands shared between the master and slave instruments is used as the evaluation index, and the genetic algorithm was used to select suitable thresholds to filter out the spectral feature information suitable for calibration transfer. The proposed SAFS algorithm was applied to two near-infrared datasets of corn oil content and larch wood density. Simultaneously, its calibration transfer performances were compared with two classical feature selection methods. The effects of different preprocessing algorithms and calibration transfer algorithms were also assessed. The model with the feature variables selected by the SAFS obtained the best prediction. The SAFS algorithm can simplify the spectral data to be transferred and improve the transfer efficiency, and the universality of the SAFS allows it to be used to optimize calibration transfer in various situations. By combining different preprocessing and classic feature selection methods with this, the sensitivity of the correlation between spectral data and component information are improved significantly, as well as the effect of calibration transfer, which will be deeply developed.

Common Latent Space Exploration for Calibration Transfer across Hyperspectral Imaging-Based Phenotyping Systems

Hyperspectral imaging has increasingly been used in high-throughput plant phenotyping systems. Rapid advancement in the field of phenotyping has resulted in a wide array of hyperspectral imaging systems. However, sharing the plant feature prediction models between different phenotyping facilities becomes challenging due to the differences in imaging environments and imaging sensors. Calibration transfer between imaging facilities is crucially important to cope with such changes. Spectral space adjustment methods including direct standardization (DS), its variants (PDS, DPDS) and spectral scale transformation (SST) require the standard samples to be imaged in different facilities. However, in real-world scenarios, imaging the standard samples is practically unattractive. Therefore, in this study, we presented three methods (TCA, c-PCA, and di-PLSR) to transfer the calibration models without requiring the standard samples. In order to compare the performance of proposed approaches, maize plants were imaged in two greenhouse-based HTPP systems using two pushbroom-style hyperspectral cameras covering the visible near-infrared range. We tested the proposed methods to transfer nitrogen content (N) and relative water content (RWC) calibration models. The results showed that prediction R2 increased by up to 14.50% and 42.20%, while the reduction in RMSEv was up to 74.49% and 76.72% for RWC and N, respectively. The di-PLSR achieved the best results for almost all the datasets included in this study, with TCA being second. The performance of c-PCA was not at par with the di-PLSR and TCA. Our results showed that the di-PLSR helped to recover the performance of RWC, and N models plummeted due to the differences originating from new imaging systems (sensor type, spectrograph, lens system, spatial resolution, spectral resolution, field of view, bit-depth, frame rate, and exposure time) or lighting conditions. The proposed approaches can alleviate the requirement of developing a new calibration model for a new phenotyping facility or to resort to the spectral space adjustment using the standard samples.

Near infrared spectral calibration model transfer without standards by screening spectral points with scale invariant feature transform from master samples spectra

In order to realize calibration model transfer of near infrared (NIR) spectra without standards, scale invariant feature transform (SIFT) algorithm was applied to extract characteristic spectral points of NIR spectra in this study. Three sets of spectral points were selected by SIFT from the spectra of precision detection (SPD) of a radix scutellariae sample by continuously testing the sample three times. Aiming at obtaining high consistency of the three sets, the orthogonal table L9 (3⁴) was used to optimize the parameters of SIFT. Basing on the NIR spectra of several representative radix scutellariae samples, a series of spectral point sets were screened by SIFT with the optimized parameters. Three methods of further treating the spectral points sets to optimize the combination of the spectral points and provided three spectral point sets, which were recorded as U_i, U_u and U_ur, respectively. The partial least square (PLS) calibration models for predicting baicalin content of radix scutellariae were built on whole wavelengths, U_i, U_u and U_ur at different number of latent variables (nLVs), respectively. Compared with other PLS models, the models of SIFT^ur-PLS built on U_ur, which was obtained by taking union of the firstly selected spectral point sets, then eliminating the points with high deviance of SPD and those with high correlativity from the union, are most robust and always give lower or lowest prediction errors for both master and slave samples at many nLVs. It is a good way to filter stable, highly independent and characteristic spectral points to build robust PLS calibration models by combining SIFT algorithm with standard deviance analysis of SPD and correlative analysis. The models can be directly shared by the slave instrument, without needing transfer sets, and without requiring to correct the spectra of slave instruments or spectral calibration models.

Modular Breath Analyzer (MBA): Introduction of a Breath Analyzer Platform Based on an Innovative and Unique, Modular eNose Concept for Breath Diagnostics and Utilization of Calibration Transfer Methods in Breath Analysis Studies

Exhaled breath analysis for early disease detection may provide a convenient method for painless and non-invasive diagnosis. In this work, a novel, compact and easy-to-use breath analyzer platform with a modular sensing chamber and direct breath sampling unit is presented. The developed analyzer system comprises a compact, low volume, temperature-controlled sensing chamber in three modules that can host any type of resistive gas sensor arrays. Furthermore, in this study three modular breath analyzers are explicitly tested for reproducibility in a real-life breath analysis experiment with several calibration transfer (CT) techniques using transfer samples from the experiment. The experiment consists of classifying breath samples from 15 subjects before and after eating a specific meal using three instruments. We investigate the possibility to transfer calibration models across instruments using transfer samples from the experiment under study, since representative samples of human breath at some conditions are difficult to simulate in a laboratory. For example, exhaled breath from subjects suffering from a disease for which the biomarkers are mostly unknown. Results show that many transfer samples of all the classes under study (in our case meal/no meal) are needed, although some CT methods present reasonably good results with only one class.

A parameter-free framework for calibration enhancement of near-infrared spectroscopy based on correlation constraint

A new parameter-free framework for calibration enhancement (PFCE) was proposed for dealing with the near-infrared (NIR) spectral inconsistency and maintaining the prediction ability of the calibration model under different conditions. The calibration issues encountered in the maintenance with or without using standards, and even the enhancement between instruments have been thoroughly addressed. The general calibration maintenance/enhancement cases were formulated into non-supervised PFCE (NS-PFCE), semi-supervised PFCE (SS-PFCE), and full-supervised PFCE (FS-PFCE). The NS-PFCE made use of both the provided master and slave spectra of standard samples to construct a maintained calibration slave model by implementing a correlation constraint on the regression coefficients. The SS-PFCE and FS-PFCE methods integrated the slave spectra and reference information of standard samples at the same time into the slave spectral calibration, and thus a maintenance or enhancement model could be achieved for the slave spectra, in particular measured on different instruments. The use of dataset1 comprised of 655 pharmaceutical tablets measured on two NIR spectrometers and datset2 containing 117 plant leaf samples in two mesh sizes has demonstrated that the PFCE framework had a significant effect on enhancing the predictions of the slave spectra in the models. The root mean square errors of prediction (RMSEPs) of either active pharmaceutical ingredient (API) amount in tablets or reducing sugar content in plant leaf samples from the slave spectra approached to or were lower than those values predicted from the master spectra in the master models established with the partial least-squares (PLS) regression method. The advantage of PFCE was parameter-free and efficient. First, the method could be flexibly employed in scientific or applicative environment with no regard to the parameter specification. Second, the performance of NS-PFCE was comparable to the classical calibration maintenance methods, yet the SS-PFCE and FS-PFCE could enhance the prediction ability to a level widely considered as the upper boundary of classical calibration maintenance methods reached.The source code of the method is available at https://github.com/JinZhangLab/PFCE.

A correlation-analysis-based wavelength selection method for calibration transfer

Considering that the spectral signals vary among different instruments, calibration transfer is required for further popularization and application of the near-infrared spectroscopy (NIRS). To achieve good calibration transfer results, spectral variables with stable and consistent signals between instruments and containing the target component information should be selected. In this study, a correlation-analysis-based wavelength selection method (CAWS) is proposed for calibration transfer. This method relies on the selection of wavelengths at which the spectral responses of master and slave instruments are well correlated (high absolute values of Pearson's correlation coefficient (|R_i|)). The proposed CAWS method was applied to two available datasets, corn and rice bran, and its calibration transfer performances were compared with other wavelength selection methods. The effects of pretreatment methods and calibration transfer algorithms were also assessed. The CAWS optimized models obtained lower root mean square errors of prediction (RMSEP_trans) after calibration transfer, suggesting that the proposed method is capable of effectively improving the efficiency of calibration transfer. Combinations of this method with other wavelength selection methods and calibration transfer algorithms may further enhance the efficiency of calibration transfer, and thus should be thoroughly investigated.

A Modified Moving-Window Partial Least-Squares Method by Coupling with Sampling Error Profile Analysis for Variable Selection in Near-Infrared Spectral Analysis

In this study, a new variable selection method, named moving-window partial least-squares coupled with sampling error profile analysis (SEPA-MWPLS), is developed. With a moving window, moving-window partial least-squares (MWPLS) is used to find window intervals which show low residual sums of squares (RSS) of a calibration set. Sampling error profile analysis (SEPA) is a useful method based on Monte-Carlo Sampling and profile analysis for cross validation (CV). By combining MWPLS with SEPA, we can obtain more stable and reliable results. Besides, we simplify the plot of the RSS line so that it is easier to determine the informative intervals. In addition, a backward elimination strategy is used to optimize the combination of subintervals. The performance of SEPA-MWPLS was tested with two near-infrared (NIR) spectra datasets and was compared with PLS, MWPLS and Monte Carlo uninformative variable elimination (MC-UVE). The results show that SEPA-MWPLS can improve model performances significantly compared with MWPLS in the number of variables, root-mean-squared errors of CV, calibration and prediction (RMSECVs, RMSECs and RMSEPs). Meanwhile it also exhibits better performances than MC-UVE.

The method of calibration model transfer by optimizing wavelength combinations based on consistent and stable spectral signals

Basing on the wavelengths with consistent and stable spectral signals between spectrometers, wavelength combinations were screened by different methods to obtain robust and simple near infrared spectra (NIR) calibration models that can be shared by slave spectrometers directly. Firstly, the wavelength set of Usc, at which the spectral signals between spectrometers are consistent and stable, was obtained by the method of screening the wavelengths with consistent and stable signals between spectrometers (SWCSS for short). Then, the wavelength set of Uscr whose spectral responses are correlated with dependent variables strongly was selected from Usc. Basing on Uscr, the methods of uninformative variable elimination (UVE), variable importance in projection (VIP) and selectivity ratio (SR) were applied to further screen optimal wavelength sets to obtain better NIR calibration models. These sets were recorded as Uscr^UVE, Uscr^VIP and Uscr^SR, respectively. The NIR partial least squares (PLS) models for predicting total alkaloids content of tobacco leaves were built on the three optimal wavelength sets, and named as Uscr^UVE-PLS, Uscr^VIP-PLS, Uscr^SR-PLS, respectively. Both Uscr^UVE-PLS and Uscr^VIP-PLS give satisfactory prediction errors for master and slave samples, and work better than the PLS model built on the whole wavelengths (WW-PLS) after piecewise direct standardization (PDS) calibration. The results show that further optimizing wavelength combinations based on consistent and stable spectral information cannot only simplify PLS models and improve the models' efficiency, but also ensure the models' accuracy when they are transferred to slave spectrometers. Wavelength selection based on the whole wavelengths without considering spectra consistency between spectrometers can improve the performance of the calibration models on the master spectrometer but cannot ensure the prediction accuracy of the slave samples.

Principal component analysis or kernel principal component analysis based joint spectral subspace method for calibration transfer

To transfer a calibration model in the case where only the master and slave spectra of standardization samples are available, principal component analysis (PCA) and kernel principal component analysis (KPCA) based joint spectral space (termed as JPCA or JKPCA) methods are proposed. As a feature subspace shared by master and slave spectra, the joint spectral subspace in JPCA and JKPCA are the projection of the joint spectral matrix comprising all the spectra of standardization by utilizing PCA and KPCA, respectively. The two corresponding low-dimensional feature matrices for master and slave spectra are extracted from the joint spectral subspace, and then a transfer matrix is estimated based on the least square criterion. In JKPCA, a partial least squares (PLS) model, named the primary model, is constructed using the low-dimensional feature matrix of master calibration spectra, and the model is then used to predict the transferred low-dimensional feature matrix of slave test spectra. Different from JKPCA, JPCA firstly reconstructs master calibration spectra and transferred slave test spectra, respectively. Then the primary model built on the reconstructed version of master calibration spectra is applied to predict the reconstructed version of transferred slave test spectra. A comparative study of the two proposed methods, multiplicative scatter correction (MSC), orthogonal signal correction (OSC), piecewise direct standardization (PDS), canonical correlation analysis based calibration transfer (CCACT), generalized least squares (GLS), slope and bias correction (SBC) and spectral space transformation (SST) is conducted on two datasets. All the statistical results together exhibit that the transfer ability of JKPCA is the best. Except JKPCA, JPCA performs at least comparable with the GLS or SST, and frequently better than the other methods.

A calibration transfer optimized single kernel near-infrared spectroscopic method

Single kernel near-infrared spectroscopy (SKNIRS) could aid in the quality screening of early-generation seeds, to improve the efficiency of seed breeding. However, the application of SKNIRS is limited due to the irregular physical characteristics, the heterogeneous constituent distributions of individual seeds, and the insufficient detection accuracy of the reference method. The reported near-infrared detection results of single seeds are often less accurate than those of dehusked seeds and seed flour. In this paper, a calibration transfer-optimized single kernel near-infrared spectroscopic method is proposed. This method aims to accurately detect the chemical composition of single seeds by using the calibration model of the corresponding dehusked seeds or seed flour. The proposed method was applied to the analysis of the protein content of a single rice kernel. The near-infrared transmission spectra of three forms of rice (single rice kernel (SRK), single brown rice kernel (SBK) and rice flour (RF)) of 201 individual rice seeds and the corresponding protein content values were obtained. By comparing different pretreatment methods and spectral ranges, the spectral range 950-1250 nm, the standard normal variate transformation (SNV) pretreatment, and 9 PLS factors were selected to construct the optimal partial least squares (PLS) regression models. Then, the protein content of single rice kernels were determined through two different methods: (i) the direct method, in which single rice kernels were analyzed using the single rice kernel model directly; and (ii) the proposed method, in which the spectra of single rice kernels were transferred into the spectra of single brown rice kernels and rice flours with a calibration transfer algorithm, spectral space transformation (SST), and were analyzed by the respective calibration models. The external validation coefficient correlation (R) value of the direct method was 0.971, and the R values of the proposed method were 0.962 (SBK) and 0.975 (RF). The root mean square error of prediction (RMSEP) value of the direct method was 0.423, and the RMSEP of the proposed method were 0.480 (SBK) and 0.401 (RF). In addition, the transfer results among the spectra of three forms of rice were compared. By comparison, the results of the proposed method are fairly close to the results of the direct method. The results indicate that the spectra generated from one individual rice seed can be transferred freely among the three forms by means of calibration transfer. The proposed method is a promising way to overcome the challenges associated with analyzing individual seeds and to improve SKNIRS.

Investigation of Direct Model Transferability Using Miniature Near-Infrared Spectrometers

Recent developments in compact near infrared (NIR) instruments, including both handheld and process instruments, have enabled easy and affordable deployment of multiple instruments for various field and online or inline applications. However, historically, instrument-to-instrument variations could prohibit success when applying calibration models developed on one instrument to additional instruments. Despite the usefulness of calibration transfer techniques, they are difficult to apply when a large number of instruments and/or a large number of classes are involved. Direct model transferability was investigated in this study using miniature near-infrared (MicroNIR™) spectrometers for both classification and quantification problems. For polymer classification, high cross-unit prediction success rates were achieved with both conventional chemometric algorithms and machine learning algorithms. For active pharmaceutical ingredient quantification, low cross-unit prediction errors were achieved with the most commonly used partial least squares (PLS) regression method. This direct model transferability is enabled by the robust design of the MicroNIR™ hardware and will make deployment of multiple spectrometers for various applications more manageable.

Calibration Transfer Based on Affine Invariance for NIR without Transfer Standards

Calibration transfer is an important field for near-infrared (NIR) spectroscopy in practical applications. However, most transfer methods are constructed with standard samples, which are expensive and difficult to obtain. Taking this problem into account, this paper proposes a calibration transfer method based on affine invariance without transfer standards (CTAI). Our method can be utilized to adjust the difference between two instruments by affine transformation. CTAI firstly establishes a partial least squares (PLS) model of the master instrument to obtain score matrices and predicted values of the two instruments, and then the regression coefficients between each of the score vectors and predicted values are computed for the master instrument and the slave instrument, respectively. Next, angles and biases are calculated between the regression coefficients of the master instrument and the corresponding regression coefficients of the slave instrument, respectively. Finally, by introducing affine transformation, new samples are predicted based on the obtained angles and biases. A comparative study between CTAI and the other five methods was conducted, and the performances of these algorithms were tested with two NIR spectral datasets. The obtained experimental results show clearly that, in general CTAI is more robust and can also achieve the best Root Mean Square Error of test sets (RMSEPs). In addition, the results of statistical difference with the Wilcoxon signed rank test show that CTAI is generally better than the others, and at least statistically the same.

Screening wavelengths with consistent and stable signals to realize calibration model transfer of near infrared spectra

Measurement environmental changes and spectral signal differences among multi-spectrometers may lead to big error in the process of transferring the model of near infrared (NIR) spectra to secondary instruments. Basing on the common sense, NIR calibration models could be shared directily among multi-instruments, if they were built on the wavelengths at which the spectral signals of secondary instruments are stable and well consistent with the primary's. Present work advanced a method named as screening wavelengths with consistent and stable signals (SWCSS) to transfer NIR calibration models. It eliminates the wavelengths at which the standard deviation of difference spectra between the primary and secondary instruments (SDDSI) is much higher than the standard deviation of precision detection spectra (SDPDS) of a sample tested on the primary and the wavelengths with higher SDPDS values. So that the spectral signals of different instruments at these selected wavelengths are consistent well and stable. The NIR calibration model is built by partial least square regression (PLS) based on the screened wavelengths. Two datasets of corn and radix scutellariae samples measured with different NIR instruments are used to test the performance of the method. The results show that the overall prediction performance of the SWCSS-PLS models for samples measured on secondaries is much better than that of the full-wavelength PLS models. The root mean square of error prediction (RMSEP) of the SWCSS-PLS models for samples tested on secondaries is equivalent or superior to that of the piecewise direct standardization (PDS) correction. The SWCSS-PLS model is of fewer parameters, robust and can give good prediction results for samples of secondary instruments. Because standard samples are no longer needed during transferring the calibration models to secondary instruments, SWCSS-PLS models could be directly shared by multi-spectroscopy instruments.

A two-level strategy for standardization of near infrared spectra by multi-level simultaneous component analysis

Standardization of near infrared (NIR) spectra is indispensable in practical applications because the spectra measured on different instruments are commonly used and the difference between the instruments must be corrected. A two-level standardization method is proposed in this study based on multi-level simultaneous component analysis (MSCA) algorithm for correcting the spectral difference between instruments. A two-level MSCA model is used to model the difference between instruments (the first level) and samples (the second level). With the two models, the spectral difference due to instruments and measurement operation can be corrected, respectively. Three NIR spectral datasets of pharmaceutical tablet, corn and plant leaf are used to evaluate the efficiency of the proposed method. The results show that the score of the first level model describes the overall spectral difference between instruments, and the score of the second level model depictures the spectral difference of the same sample between the measurements. The latter difference may include the spectral variations caused by instrument, operation and the measurement conditions. Therefore, both the spectral difference due to the instrument and measurement can be corrected by adjusting the coefficients in the scores of the two level models, respectively. The proposed method provides a good way for standardizing the spectra measured on different instruments when the measurement is not reproducible.

Eliminating Non-linear Raman Shift Displacement Between Spectrometers via Moving Window Fast Fourier Transform Cross-Correlation

Obtaining consistent spectra by using different spectrometers is of critical importance to the fields that rely heavily on Raman spectroscopy. The quality of both qualitative and quantitative analysis depends on the stability of specific Raman peak shifts across instruments. Non-linear drifts in the Raman shifts can, however, introduce additional complexity in model building, potentially even rendering a model impractical. Fortunately, various types of shift correction methods can be applied in data preprocessing in order to address this problem. In this work, a moving window fast Fourier transform cross-correlation is developed to correct non-linear shifts for synchronization of spectra obtained from different Raman instruments. The performance of this method is demonstrated by using a series of Raman spectra of pharmaceuticals as well as comparing with data obtained by using an existing standard Raman shift scattering procedure. The results show that after the removal of shift displacements, the spectral consistency improves significantly, i.e., the spectral correlation coefficient of the two Raman instruments increased from 0.87 to 0.95. The developed standardization method has, to a certain extent, reduced instrumental systematic errors caused by measurement, while enhancing spectral compatibility and consistency through a simple and flexible moving window procedure.

Multi-spectrometer calibration transfer based on independent component analysis

Calibration transfer is indispensable for practical applications of near infrared (NIR) spectroscopy due to the need for precise and consistent measurements across different spectrometers. In this work, a method for multi-spectrometer calibration transfer is described based on independent component analysis (ICA). A spectral matrix is first obtained by aligning the spectra measured on different spectrometers. Then, by using independent component analysis, the aligned spectral matrix is decomposed into the mixing matrix and the independent components of different spectrometers. These differing measurements between spectrometers can then be standardized by correcting the coefficients within the independent components. Two NIR datasets of corn and edible oil samples measured with three and four spectrometers, respectively, were used to test the reliability of this method. The results of both datasets reveal that spectra measurements across different spectrometers can be transferred simultaneously and that the partial least squares (PLS) models built with the measurements on one spectrometer can predict that the spectra can be transferred correctly on another.

Domain-Invariant Partial-Least-Squares Regression

Multivariate calibration models often fail to extrapolate beyond the calibration samples because of changes associated with the instrumental response, environmental condition, or sample matrix. Most of the current methods used to adapt a source calibration model to a target domain exclusively apply to calibration transfer between similar analytical devices, while generic methods for calibration-model adaptation are largely missing. To fill this gap, we here introduce domain-invariant partial-least-squares (di-PLS) regression, which extends ordinary PLS by a domain regularizer in order to align the source and target distributions in the latent-variable space. We show that a domain-invariant weight vector can be derived in closed form, which allows the integration of (partially) labeled data from the source and target domains as well as entirely unlabeled data from the latter. We test our approach on a simulated data set where the aim is to desensitize a source calibration model to an unknown interfering agent in the target domain (i.e., unsupervised model adaptation). In addition, we demonstrate unsupervised, semisupervised, and supervised model adaptation by di-PLS on two real-world near-infrared (NIR) spectroscopic data sets.

Mutual factor analysis for quantitative analysis by temperature dependent near infrared spectra

Temperature dependent near infrared (NIR) spectroscopy has been developed for analyzing multi-component mixtures and understanding the molecular interactions in solutions. In this work, a chemometric method named as mutual factor analysis (MFA) was proposed for the analysis of temperature dependent NIR spectra. The method extracts the common spectral feature contained in the spectra of different temperature or different concentration. The relative quantity of the extracted spectral feature is proportional to the temperature or concentration. From the spectra of water-glucose mixtures, both the spectral variations induced by temperature and concentration are obtained and the variations are correlated with the inducements, respectively, in a very good linearity. Serum samples were used for validation of the method. An acceptable calibration model with a good correlation coefficient (R² = 0.8639) was obtained for glucose measurement. The relative deviations of the measured concentrations from the calibration model are in the range of -18.7-8.52%, which are in a reasonable level for clinical uses. More importantly, the calculations are based on the spectral information of water that has interactions with the analyte. This provides a new way for quantitative analyses of bio-systems.

Additive Partial Least Squares for efficient modelling of independent variance sources demonstrated on practical case studies

A model recalibration method based on additive Partial Least Squares (PLS) regression is generalized for multi-adjustment scenarios of independent variance sources (referred to as additive PLS - aPLS). aPLS allows for effortless model readjustment under changing measurement conditions and the combination of independent variance sources with the initial model by means of additive modelling. We demonstrate these distinguishing features on two NIR spectroscopic case-studies. In case study 1 aPLS was used as a readjustment method for an emerging offset. The achieved RMS error of prediction (1.91 a.u.) was of similar level as before the offset occurred (2.11 a.u.). In case-study 2 a calibration combining different variance sources was conducted. The achieved performance was of sufficient level with an absolute error being better than 0.8% of the mean concentration, therefore being able to compensate negative effects of two independent variance sources. The presented results show the applicability of the aPLS approach. The main advantages of the method are that the original model stays unadjusted and that the modelling is conducted on concrete changes in the spectra thus supporting efficient (in most cases straightforward) modelling. Additionally, the method is put into context of existing machine learning algorithms.

Optimization of rice amylose determination by NIR-spectroscopy using PLS chemometrics algorithms

Determining amylose content in rice with near infrared (NIR) spectroscopy, associated with a suitable multivariate regression method, is both feasible and relevant for the rice business to enable Process Analytical Technology applications for this critical factor, but it has not been fully exploited. Due to it being time-consuming and prone to experimental errors, it is urgent to develop a low-cost, nondestructive and 'on-line' method able to provide high accuracy and reproducibility. Different rice varieties and specific chemometrics tools, such as partial least squares (PLS), interval-PLS, synergy interval-PLS and moving windows-PLS, were applied to develop an optimal regression model for rice amylose determination. The model performance was evaluated by the root mean square error of prediction (RMSEP) and the correlation coefficient (R). The high performance of the siPLS method (R=0.94; RMSEP=1.938; 8941-8194cm^-1; 5592-5045cm^-1; and 4683-4335cm^-1) shows the feasibility of NIR technology for determination of the amylose with high accuracy.

Calibration transfer of near-infrared spectroscopy by canonical correlation analysis coupled with wavelet transform

Calibration model transfer has played a prominent role in the practical application of NIR spectral analysis. The change of instruments and sample physical states may lead to variation of the NIR spectrum, which results in the applicability of the model in judicatory practice being unsatisfactory. Therefore, a transfer for the calibration model considering both the variation of instruments and sample states is a necessity to ensure its availability. In this paper, a novel approach, namely canonical correlation analysis coupled with wavelet transform (WTCCA), was proposed for calibration transfer between two near infrared spectrometers (a portable and a laboratory instrument), and simultaneously, among three physical states (tobacco powder, tobacco filament and intact leaf) to determine the content of total sugars, reducing sugars, and nicotine in tobacco leaf samples, respectively. Wavelet transform (WT) is introduced to reduce noise and deduct background shifts from the spectra by compression, and then, calibration transfer by canonical correlation analysis (CTCCA) extracts the compressed spectral similarities using canonical scores for spectra correction. Three similar standardization algorithms, including piecewise direct standardization (PDS), piecewise direct standardization with wavelet transform (WTPDS), and CTCCA were compared with WTCCA to evaluate its relative performance. The obtained results showed that the employment of WTCCA yielded the lowest root mean standard error of prediction (RMSEP) on the three analytes in three physical states. For the tobacco powder dataset, the RMSEP values had a reduction of 25.83%, 13.96%, and 14.22% compared with the values of direct prediction without spectra transfer, respectively. For the tobacco filament dataset, the corresponding values were decreased by 18.06%, 14.90%, and 13.61% and for the intact leaf dataset, the values had dropped by 10.70%, 18.21%, and 28.21%, respectively. In summary, the comprehensive investigation carried out in the present work shows that WTCCA is very appropriate for correcting the variations caused by the change of machines and sample states. Furthermore, WTCCA is a promising calibration transfer method which can be recommended for on-line/in-line application.

Linear model correction: A method for transferring a near-infrared multivariate calibration model without standard samples

Calibration transfer is essential for practical applications of near infrared (NIR) spectroscopy because the measurements of the spectra may be performed on different instruments and the difference between the instruments must be corrected. For most of calibration transfer methods, standard samples are necessary to construct the transfer model using the spectra of the samples measured on two instruments, named as master and slave instrument, respectively. In this work, a method named as linear model correction (LMC) is proposed for calibration transfer without standard samples. The method is based on the fact that, for the samples with similar physical and chemical properties, the spectra measured on different instruments are linearly correlated. The fact makes the coefficients of the linear models constructed by the spectra measured on different instruments are similar in profile. Therefore, by using the constrained optimization method, the coefficients of the master model can be transferred into that of the slave model with a few spectra measured on slave instrument. Two NIR datasets of corn and plant leaf samples measured with different instruments are used to test the performance of the method. The results show that, for both the datasets, the spectra can be correctly predicted using the transferred partial least squares (PLS) models. Because standard samples are not necessary in the method, it may be more useful in practical uses.

Transfer of multivariate regression models between high-resolution NMR instruments: application to authenticity control of sunflower lecithin

In recent years the number of spectroscopic studies utilizing multivariate techniques and involving different laboratories has been dramatically increased. In this paper the protocol for calibration transfer of partial least square regression model between high-resolution nuclear magnetic resonance (NMR) spectrometers of different frequencies and equipped with different probes was established. As the test system previously published quantitative model to predict the concentration of blended soy species in sunflower lecithin was used. For multivariate modelling piecewise direct standardization (PDS), direct standardization, and hybrid calibration were employed. PDS showed the best performance for estimating lecithin falsification regarding its vegetable origin resulting in a significant decrease in root mean square error of prediction from 5.0 to 7.3% without standardization to 2.9-3.2% for PDS. Acceptable calibration transfer model was obtained by direct standardization, but this standardization approach introduces unfavourable noise to the spectral data. Hybrid calibration is least recommended for high-resolution NMR data. The sensitivity of instrument transfer methods with respect to the type of spectrometer, the number of samples and the subset selection was also discussed. The study showed the necessity of applying a proper standardization procedure in cases when multivariate model has to be applied to the spectra recorded on a secondary NMR spectrometer even with the same magnetic field strength. Copyright © 2016 John Wiley & Sons, Ltd.