Open-source python module for automated preprocessing of near infrared spectroscopic data

Fusion of convolutional neural network with XGBoost feature extraction for predicting multi-constituents in corn using near infrared spectroscopy

Near-infrared (NIR) spectroscopy has been widely utilized to predict multi-constituents of corn in agriculture. However, directly extracting constituent information from the NIR spectra is challenging due to many issues such as broad absorption band, overlapping and non-specific nature. To solve these problems and extract implicit features from the raw data of NIR spectra to improve performance of quantitative models, a one-dimensional shallow convolutional neural network (CNN) model based on an eXtreme Gradient Boosting (XGBoost) feature extraction method was proposed in this paper. The leaf node feature information in the XGBoost was encoded and reconstructed to obtain the implicit features of raw data in the NIR spectra. A two-parametric Swish (TSwish or TS) activation function was proposed to improve the performance of CNN, and the elastic net (EN) was also applied to avoid the overfitting problem of the CNN model. Performance of the developed XGBoost-CNN-TS-EN model was evaluated using two public NIR spectroscopy datasets of corn and soil, and the obtained determination coefficients (R2) for moisture, oil, protein, and starch of the corn on test set were 0.993, 0.991, 0.998, and 0.992, respectively, with that of the soil organic matter being 0.992. The XGBoost-CNN-TS-EN model exhibits superior stability, good prediction accuracy, and generalization ability, demonstrating its great potentials for quantitative analysis of multi-constituents in spectroscopic applications.

XGBoost algorithm assisted multi-component quantitative analysis with Raman spectroscopy

To improve prediction performance and reduce artifacts in Raman spectra, we developed an eXtreme Gradient Boosting (XGBoost) preprocessing method to preprocess the Raman spectra of glucose, glycerol and ethanol mixtures. To ensure the robustness and reliability of the XGBoost preprocessing method, an X-LR model (which combined XGBoost preprocessing and a linear regression (LR) model) and a X-MLP model (which combined XGBoost preprocessing and a multilayer perceptron (MLP) model) were developed. These two models were used to quantitatively analyze the concentrations of glucose, glycerol and ethanol in the Raman spectra of mixed solutions. The proportion map of hyperparameters was firstly used to narrow down the search range of hyperparameters in the X-LR and the X-MLP models. Then the correlation coefficients (R2), root mean square of calibration (RMSEC), and root mean square error of prediction (RMSEP) were used to evaluate the models' performance. Experimental results indicated that the XGBoost preprocessing method achieved higher accuracy and generalization capability, with better performance than those of other preprocessing methods for both LR and MLP models.

Development of automatic tuning for combined preprocessing and hyperparameters of machine learning and its application to NIR spectral data of coconut milk adulteration

This study proposed a novel approach to automatically select the preprocessing methods and hyperparameters of machine learning (ML) algorithms based on their best performance in cross-validation for near-infrared (NIR) spectroscopy data. The proposed method simultaneously incorporates single or multiple-preprocessing steps and tunes hyperparameters to determine the best model performance for FT-NIR and Micro-NIR spectral data of coconut milk adulteration with distilled water and mature coconut water in the range of 0%-50%. Computational experiments were conducted using nine single preprocessing types, three types of ML classifier (linear discriminant analysis (LDA), k-nearest neighbour (KNN), multilayer perceptron (MLP)) and three types of ML regressor (partial least squares (PLS), KNN, MLP). The proposed performance strategy effectively addressed and produced satisfactory outcomes for classification and regression challenges in coconut milk adulteration. Finally, the results demonstrated that the proposed approach can more accurately determine the best model, particularly for NIR spectroscopy of coconut milk adulteration.

Chemometrics and analytical blank on the at-line monitoring of Zika-VLP production using near-infrared spectroscopy

The Zika disease caused by the Zika virus was declared a Public Health Emergency by the World Health Union (WHO), with microcephaly as the most critical consequence. Aiming to reduce the spread of the virus, biopharmaceutical organizations invest in vaccine research and production, based on multiple platforms. A crescent vaccine production approach is based on virus-like particles (VLP), for not having genetic material in its composition, hypoallergenic and non-mutant character. For bioprocess, it is essential to have means of real-time monitoring, which can be assessed using process analysis techniques such as Near-infrared (NIR) spectroscopy, that can be combined with chemometric methods, like Partial-Least Squares (PLS) and Artificial Neural Networks (ANN) for prediction of biochemical variables. This work proposes a biochemical Zika VLP upstream production at-line monitoring model using NIR spectroscopy comparing sampling conditions (with or without cells), analytical blank (air, ultrapure water), and spectra pre-processing approaches. Seven experiments in a benchtop bioreactor using recombinant baculovirus/Sf9 insect cell platform in serum-free medium were performed to obtain biochemical and spectral data for chemometrics modeling (PLS and ANN), composed by a random data split (80 % calibration, 20 % validation) for cross-validation of the PLS models and 70 % training, 15 % testing, 15 % validation for ANN. The best models generated in the present work presented an average absolute error of 1.59 × 105 cell/mL for density of viable cells, 2.37 % for cell viability, 0.25 g/L for glucose, 0.007 g/L for lactate, 0.138 g/L for glutamine, 0.18 g/L for glutamate, 0,003 g/L for ammonium, and 0.014 g/L for potassium.

Rapid and accurate quality evaluation of Angelicae Sinensis Radix based on near-infrared spectroscopy and Bayesian optimized LSTM network

The authentic traditional Chinese medicines (TCMs) including Angelicae Sinensis Radix (ASR) are the representative of high-quality herbals in China. However, ASR from authentic region being adulterated or counterfeited is frequently occurring, and there is still a lack of rapid quality evaluation methods for identifying the authentic ASR. In this study, the color features of ASR were firstly characterized. The results showed that the authentic ASR cannot be fully identified by color characteristics. Then near-infrared (NIR) spectroscopy combined with Bayesian optimized long short-term memory (BO-LSTM) was used to evaluate the quality of ASR, and the performance of BO-LSTM with common classification and regression algorithms was compared. The results revealed that following the pretreatment of NIR spectra, the optimal NIR spectra combined with BO-LSTM not only successfully distinguished authentic, non-authentic, and adulterated ASR with 100 % accuracy, but also accurately predicted the adulteration concentration of authentic ASR (R2 > 0.99). Moreover, BO-LSTM demonstrated excellent performance in classification and regression compared with common algorithms (ANN, SVM, PLSR, etc.). Overall, the proposed strategy could quickly and accurately evaluate the quality of ASR, which provided a reference for other TCMs.

NIR spectroscopy-CNN-enabled chemometrics for multianalyte monitoring in microbial fermentation

As the biopharmaceutical industry looks to implement Industry 4.0, the need for rapid and robust analytical characterization of analytes has become a pressing priority. Spectroscopic tools, like near-infrared (NIR) spectroscopy, are finding increasing use for real-time quantitative analysis. Yet detection of multiple low-concentration analytes in microbial and mammalian cell cultures remains an ongoing challenge, requiring the selection of carefully calibrated, resilient chemometrics for each analyte. The convolutional neural network (CNN) is a puissant tool for processing complex data and making it a potential approach for automatic multivariate spectral processing. This work proposes an inception module-based two-dimensional (2D) CNN approach (I-CNN) for calibrating multiple analytes using NIR spectral data. The I-CNN model, coupled with orthogonal partial least squares (PLS) preprocessing, converts the NIR spectral data into a 2D data matrix, after which the critical features are extracted, leading to model development for multiple analytes. Escherichia coli fermentation broth was taken as a case study, where calibration models were developed for 23 analytes, including 20 amino acids, glucose, lactose, and acetate. The I-CNN model result statistics depicted an average R2 values of prediction 0.90, external validation data set 0.86 and significantly lower root mean square error of prediction values ∼0.52 compared to conventional regression models like PLS. Preprocessing steps were applied to I-CNN models to evaluate any augmentation in prediction performance. Finally, the model reliability was assessed via real-time process monitoring and comparison with offline analytics. The proposed I-CNN method is systematic and novel in extracting distinctive spectral features from a multianalyte bioprocess data set and could be adapted to other complex cell culture systems requiring rapid quantification using spectroscopy.

Classification of the Crosslink Density Level of Para Rubber Thick Film of Medical Glove by Using Near-Infrared Spectral Data

Classification of the crosslink density level of para rubber medical gloves by using near-infrared spectral data combined with machine learning is the first time reported in this paper. The spectra of medical glove samples with different crosslink densities acquired by an ultra-compact portable MicroNIR spectrometer were correlated with their crosslink density levels, which were referencely evaluated by the toluene swell index (TSI). The machine learning protocols used to classify the 3 groups of TSI were specified as less than 80% TSI, 80-88% TSI, and more than 88% TSI. The 80-88% TSI group was the group in which the compounded latex was suitable for medical glove production, which made the glove specification comply with the requirements of customers as indicated by the tensile test. The results show that when comparing the algorithms used for modeling, the linear discriminant analysis (LDA) developed by 2nd derivative spectra with 15 k-best selected wavelengths fairly accurately predicted the class but was most reliable among other algorithms, i.e., artificial neural networks (ANN), support vector machines (SVM), and k-nearest neighbors (kNN), due to higher prediction accuracy, precision, recall, and F1-score of the same value of 0.76 and no overfitting or underfitting prediction. This developed model can be implemented in the glove factory for screening purposes in the production line. However, deep learning modeling should be explored with a larger sample number required for better model performance.

SVR Chemometrics to Quantify β-Lactoglobulin and α-Lactalbumin in Milk Using MIR

Protein content variation in milk can impact the quality and consistency of dairy products, necessitating access to in-line real time monitoring. Here, we present a chemometric approach for the qualitative and quantitative monitoring of β-lactoglobulin and α-lactalbumin, using mid-infrared spectroscopy (MIR). In this study, we employed Hotelling T2 and Q-residual for outlier detection, automated preprocessing using nippy, conducted wavenumber selection with genetic algorithms, and evaluated four chemometric models, including partial least squares, support vector regression (SVR), ridge, and logistic regression to accurately predict the concentrations of β-lactoglobulin and α-lactalbumin in milk. For the quantitative analysis of these two whey proteins, SVR performed the best to interpret protein concentration from 197 MIR spectra originating from 42 Cornell University samples of preserved pasteurized modified milk. The R2 values obtained for β-lactoglobulin and α-lactalbumin using leave one out cross-validation (LOOCV) are 92.8% and 92.7%, respectively, which is the highest correlation reported to date. Our approach introduced a combination of preprocessing automation, genetic algorithm-based wavenumber selection, and used Optuna to optimize the framework for tuning hyperparameters of the chemometric models, resulting in the best chemometric analysis of MIR data to quantitate β-lactoglobulin and α-lactalbumin to date.

Self-expansion full information optimization strategy: Convenient and efficient method for near infrared spectrum auto-analysis

An essential step in the application of near infrared spectroscopy technology is the spectrum preprocessing. A reasonable implementation of it ensures that the effective spectral information is correctly extracted and, also that the model's accuracy is increased. However, some analysts' research still uses the manual approach of trial and error, particularly those less skilled ones. Previous papers have provided preprocessing optimization algorithms for NIR, but there are still some problems that need to be resolved, such as the unwieldy sequence determination of preprocessing method or, the fluctuated optimization outcomes or, lack of sufficient statistical information. This research suggests a spectrum auto-analysis methodology named self-expansion full information optimization strategy, a new powerful open-source technique for concurrently addressing all of these above issues simultaneously. For the first time in the field of chemometrics, this algorithm offers a reliable and effective automatic near infrared auto-modelling method based on the statistical informatics. With the aid of its built-in modules, such as information generators, spectrum processors, etc., it is able to fully search the common preprocessing techniques, which is determined by Monte Carlo cross validation. Then the final ensemble calibration model is built by employing the optimized preprocessing schemes, along with the wavelength variables screening algorithm. The optimization strategy can offer the user objective useful statistics information created throughout the modeling process to further examine the model's effectiveness. The results demonstrate that the suggested method can easily and successfully extract spectrum information and develop calibration models by putting it to the test on two groups of actual near-infrared spectral data. Additionally, this optimization strategy can also be applied to other spectrum analysis areas, such Raman spectroscopy or infrared spectroscopy, by changing a few of its parameters, and has extraordinary application value.

ATR-FTIR spectroscopy combined with chemometrics to assess the spectral markers of irradiated baijius and their potential application in irradiation dose control

Although some methods have been proposed for the identification of irradiated baijius, they are often costly, time-consuming, and destructive. It is also unclear what instrumentation can be used to fully characterize the quality changes in irradiated baijius. To address this issue, this study pioneers the use of attenuated total reflection Fourier transform infrared (ATR-FTIR) spectroscopy in combination with chemometrics to open up new avenues for characterizing irradiated baijius and their quality control. Principal component analysis, five spectral pre-processing methods (Savitzky-Golay smoothing (S-G), second-order derivative (SD), multiple scattering correction (MSC), S-G + SD and S-G + MSC), five wavelength selection methods (random forest variable importance (RFVI), two-dimensional correlation spectroscopy (2D-COS), variable importance in projection (VIP), ReliefF, and Venn), and three classification models (partial least squares-discriminant analysis (PLS-DA), random forest (RF), and grasshopper optimization algorithm-based support vector machine (GOA-SVM)) were integrated into the analytical framework of ATR-FTIR spectroscopy, aiming to accurately identify baijiu samples according to different irradiation doses and to search for irradiation-induced spectral difference characteristics (spectral markers). The results showed that SD was the best spectral pre-processing method, and RF models constructed using the 20 most competitive and discriminative spectral markers (selected by Venn) could achieve accurate identification of baijiu samples based on irradiation dose (0, 4, 6, and 8 kGy). After Pearson correlation analysis, the five significantly (P＜0.05) changed spectral markers (1596, 2025, 2309, 2329, and 2380 cm-1) were attributed to changes in the content of total acids, alcohols, and aromatic compounds. These findings demonstrate for the first time the potential of ATR-FTIR spectroscopy as a fast, low-cost, and non-destructive tool for the characterization and identification of irradiated baijiu samples. This approach may also offer a promising solution for labeling management of irradiated foods, vintage identification of baijius, and brand protection.

Towards a point-of-care multimodal spectroscopy instrument for the evaluation of human cardiac tissue

To demonstrate that point-of-care multimodal spectroscopy using Near-Infrared (NIR) and Raman Spectroscopy (RS) can be used to diagnose human heart tissue. We generated 105 spectroscopic scans, which comprised 4 NIR and 3 RS scans per sample to generate a "multimodal spectroscopic scan" (MSS) for each heart, done across 15 patients, 5 each from the dilated cardiomyopathy (DCM), Ischaemic Heart Disease (IHD) and Normal pathologies. Each of the MSS scans was undertaken in 3 s. Data were entered into machine learning (ML) algorithms to assess accuracy of MSS in diagnosing tissue type. The median age was 50 years (IQR 49-52) for IHD, 47 (IQR 45-50) for DCM and 36 (IQR 33-52) for healthy patients (p = 0.35), 60% of which were male. MSS identified key differences in IHD, DCM and normal heart samples in regions typically associated with fibrosis and collagen (NIR wavenumbers: 1433, 1509, 1581, 1689 and 1725 nm; RS wavelengths: 1658, 1450 and 1330 cm-1). In principal component (PC) analyses, these differences explained 99.2% of the variation in 4 PCs for NIR, 81.6% in 10 PCs for Raman, and 99.0% in 26 PCs for multimodal spectroscopic signatures. Using a stack machine learning algorithm with combined NIR and Raman data, our model had a precision of 96.9%, recall of 96.6%, specificity of 98.2% and Area Under Curve (AUC) of 0.989 (Table 1). NIR and Raman modalities alone had similar levels of precision at 94.4% and 89.8% respectively (Table 1). MSS combined with ML showed accuracy of 90% for detecting dilated cardiomyopathy, 100% for ischaemic heart disease and 100% for diagnosing healthy tissue. Multimodal spectroscopic signatures, based on NIR and Raman spectroscopy, could provide cardiac tissue scans in 3-s to aid accurate diagnoses of fibrosis in IHD, DCM and normal hearts. Table 1 Machine learning performance metrics for validation data sets of (a) Near-Infrared (NIR), (b) Raman and (c and d) multimodal data using logistic regression (LR), stochastic gradient descent (SGD) and support vector machines (SVM), with combined "stack" (LR + SGD + SVM) AUC Precision Recall Specificity (a) NIR model  Logistic regression 0.980 0.944 0.933 0.967  SGD 0.550 0.281 0.400 0.700  SVM 0.840 0.806 0.800 0.900  Stack 0.933 0.794 0.800 0.900 (b) Raman model  Logistic regression 0.985 0.940 0.929 0.960  SGD 0.892 0.869 0.857 0.932  SVM 0.992 0.940 0.929 0.960  Stack 0.954 0.869 0.857 0.932 (c) MSS: multimodal (NIR + Raman) to detect DCM vs. IHD vs. normal patients  Logistic regression 0.975 0.841 0.828 0.917  SGD 0.847 0.803 0.793 0.899  SVM 0.971 0.853 0.828 0.917  Stack 0.961 0.853 0.828 0.917 (d) MSS: multimodal (NIR + Raman) to detect pathological vs. normal patients  Logistic regression 0.961 0.969 0.966 0.984  SGD 0.944 0.967 0.966 0.923  SVM 1.000 1.000 1.000 1.000  Stack 1.000 0.944 0.931 0.969 Bold values indicate values obtained from the stack algorithm and used for analyses.

Point-of-care detection of fibrosis in liver transplant surgery using near-infrared spectroscopy and machine learning

Introduction

Visual assessment and imaging of the donor liver are inaccurate in predicting fibrosis and remain surrogates for histopathology. We demonstrate that 3-s scans using a handheld near-infrared-spectroscopy (NIRS) instrument can identify and quantify fibrosis in fresh human liver samples.

Methods

We undertook NIRS scans on 107 samples from 27 patients, 88 from 23 patients with liver disease, and 19 from four organ donors.

Results

Liver disease patients had a median immature fibrosis of 40% (interquartile range [IQR] 20-60) and mature fibrosis of 30% (10%-50%) on histopathology. The organ donor livers had a median fibrosis (both mature and immature) of 10% (IQR 5%-15%). Using machine learning, this study detected presence of cirrhosis and METAVIR grade of fibrosis with a classification accuracy of 96.3% and 97.2%, precision of 96.3% and 97.0%, recall of 96.3% and 97.2%, specificity of 95.4% and 98.0% and area under receiver operator curve of 0.977 and 0.999, respectively. Using partial-least square regression machine learning, this study predicted the percentage of both immature (R 2 = 0.842) and mature (R 2 = 0.837) with a low margin of error (root mean square of error of 9.76% and 7.96%, respectively).

Conclusion

This study demonstrates that a point-of-care NIRS instrument can accurately detect, quantify and classify liver fibrosis using machine learning.

Visible and Near-Infrared Spectroscopy Enables Differentiation of Normal and Early Osteoarthritic Human Knee Joint Articular Cartilage

Osteoarthritis degenerates cartilage and impairs joint function. Early intervention opportunities are missed as current diagnostic methods are insensitive to early tissue degeneration. We investigated the capability of visible light-near-infrared spectroscopy (Vis-NIRS) to differentiate normal human cartilage from early osteoarthritic one. Vis-NIRS spectra, biomechanical properties and the state of osteoarthritis (OARSI grade) were quantified from osteochondral samples harvested from different anatomical sites of human cadaver knees. Two support vector machines (SVM) classifiers were developed based on the Vis-NIRS spectra and OARSI scores. The first classifier was designed to distinguish normal (OARSI: 0-1) from general osteoarthritic cartilage (OARSI: 2-5) to check the general suitability of the approach yielding an average accuracy of 75% (AUC = 0.77). Then, the second classifier was designed to distinguish normal from early osteoarthritic cartilage (OARSI: 2-3) yielding an average accuracy of 71% (AUC = 0.73). Important wavelength regions for differentiating normal from early osteoarthritic cartilage were related to collagen organization (wavelength region: 400-600 nm), collagen content (1000-1300 nm) and proteoglycan content (1600-1850 nm). The findings suggest that Vis-NIRS allows objective differentiation of normal and early osteoarthritic tissue, e.g., during arthroscopic repair surgeries.

Characterisation of Cartilage Damage via Fusing Mid-Infrared, Near-Infrared, and Raman Spectroscopic Data

Mid-infrared spectroscopy (MIR), near-infrared spectroscopy (NIR), and Raman spectroscopy are all well-established analytical techniques in biomedical applications. Since they provide complementary chemical information, we aimed to determine whether combining them amplifies their strengths and mitigates their weaknesses. This study investigates the feasibility of the fusion of MIR, NIR, and Raman spectroscopic data for characterising articular cartilage integrity. Osteochondral specimens from bovine patellae were subjected to mechanical and enzymatic damage, and then MIR, NIR, and Raman data were acquired from the damaged and control specimens. We assessed the capacity of individual spectroscopic methods to classify the samples into damage or control groups using Partial Least Squares Discriminant Analysis (PLS-DA). Multi-block PLS-DA was carried out to assess the potential of data fusion by combining the dataset by applying two-block (MIR and NIR, MIR and Raman, NIR and Raman) and three-block approaches (MIR, NIR, and Raman). The results of the one-block models show a higher classification accuracy for NIR (93%) and MIR (92%) than for Raman (76%) spectroscopy. In contrast, we observed the highest classification efficiency of 94% and 93% for the two-block (MIR and NIR) and three-block models, respectively. The detailed correlative analysis of the spectral features contributing to the discrimination in the three-block models adds considerably more insight into the molecular origin of cartilage damage.

Near-infrared spectroscopy for structural bone assessment

Aims

Disorders of bone integrity carry a high global disease burden, frequently requiring intervention, but there is a paucity of methods capable of noninvasive real-time assessment. Here we show that miniaturized handheld near-infrared spectroscopy (NIRS) scans, operated via a smartphone, can assess structural human bone properties in under three seconds.

Methods

A hand-held NIR spectrometer was used to scan bone samples from 20 patients and predict: bone volume fraction (BV/TV); and trabecular (Tb) and cortical (Ct) thickness (Th), porosity (Po), and spacing (Sp).

Results

NIRS scans on both the inner (trabecular) surface or outer (cortical) surface accurately identified variations in bone collagen, water, mineral, and fat content, which then accurately predicted bone volume fraction (BV/TV, inner R² = 0.91, outer R² = 0.83), thickness (Tb.Th, inner R² = 0.9, outer R² = 0.79), and cortical thickness (Ct.Th, inner and outer both R² = 0.90). NIRS scans also had 100% classification accuracy in grading the quartile of bone thickness and quality.

Conclusion

We believe this is a fundamental step forward in creating an instrument capable of intraoperative real-time use.

Cite this article: Bone Jt Open 2023;4(4):250–261.

Nondestructive Evaluation of Mechanical and Histological Properties of the Human Aorta With Near-Infrared Spectroscopy

INTRODUCTION

Ascending aortic dilatation is a well-known risk factor for aortic rupture. Indications for aortic replacement in its dilatation concomitant to other open-heart surgery exist; however, cut-off values based solely on aortic diameter may fail to identify patients with weakened aortic tissue. We introduce near-infrared spectroscopy (NIRS) as a diagnostic tool to nondestructively evaluate the structural and compositional properties of the human ascending aorta during open-heart surgeries. During open-heart surgery, NIRS could provide information regarding tissue viability in situ and thus contribute to the decision of optimal surgical repair.

MATERIALS AND METHODS

Samples were collected from patients with ascending aortic aneurysm (n = 23) undergoing elective aortic reconstruction surgery and from healthy subjects (n = 4). The samples were subjected to spectroscopic measurements, biomechanical testing, and histological analysis. The relationship between the near-infrared spectra and biomechanical and histological properties was investigated by adapting partial least squares regression.

RESULTS

Moderate prediction performance was achieved with biomechanical properties (r = 0.681, normalized root-mean-square error of cross-validation = 17.9%) and histological properties (r = 0.602, normalized root-mean-square error of cross-validation = 22.2%). Especially the performance with parameters describing the aorta's ultimate strength, for example, failure strain (r = 0.658), and elasticity (phase difference, r = 0.875) were promising and could, therefore, provide quantitative information on the rupture sensitivity of the aorta. For the estimation of histological properties, the results with α-smooth muscle actin (r = 0.581), elastin density (r = 0.973), mucoid extracellular matrix accumulation(r = 0.708), and media thickness (r = 0.866) were promising.

CONCLUSIONS

NIRS could be a potential technique for in situ evaluation of biomechanical and histological properties of human aorta and therefore useful in patient-specific treatment planning.

Nondestructive Detection of Moisture Content in Palm Oil by Using Portable Vibrational Spectroscopy and Optimal Prediction Algorithms

Rapid and nondestructive measurement of moisture content in crude palm oil is essential for promoting the shelf-stability and quality. In this research, micro NIR spectrometer coupled with a multivariate calibration model was used to collect and analyse fingerprinted information from palm oil samples at different moisture contents. Several preprocessing methods such as standard normal variant (SNV), multiplicative scatter correction (MSC), Savitzky-Golay first derivative (SGD1), Savitzky-Golay second derivative (SGD2) together with partial least square (PLS) regression techniques, full PLS, interval PLS (iPLS), synergy interval PLS (SiPLS), genetic algorithm PLS (GAPLS), and successive projection algorithm PLS (SPA-PLS) were comparatively employed to construct an optimum quantitative prediction model for moisture content in crude palm oil. The models were evaluated according to the coefficient of determination and root mean square error in calibration (Rc and RMSEC) and prediction (Rp and RMSEC) set, respectively. The model SGD1 + SiPLS was the optimal novel algorithm obtained among the others with the performance of Rc = 0.968 and RMSEC = 0.468 in the calibration set and Rp = 0.956 and RMSEP = 0.361 in the prediction set. The results showed that rapid and nondestructive determination of moisture content in palm oil is feasible and this would go a long way to facilitating quality control of crude palm oil.

Navigating with chemometrics and machine learning in chemistry

Chemometrics and machine learning are artificial intelligence-based methods stirring a transformative change in chemistry. Organic synthesis, drug discovery and analytical techniques are incorporating machine learning techniques at an accelerated pace. However, machine-assisted chemistry faces challenges while solving critical problems in chemistry due to complex relationships in data sets. Even with increasing publishing volumes on machine learning, its application in areas of chemistry is not a straightforward endeavour. A particular concern in applying machine learning in chemistry is data availability and reproducibility. The present review article discusses the various chemometric methods, expert systems, and machine learning techniques developed for solving problems of organic synthesis and drug discovery with selected examples. Further, a concise discussion on chemometrics and ML deployed in analytical techniques such as, spectroscopy, microscopy and chromatography are presented. Finally, the review reflects the challenges, opportunities and future perspectives on machine learning and automation in chemistry. The review concludes by pondering on some tough questions on applying machine learning and their possibility of navigation in the different terrains of chemistry.

Structural Analysis and Classification of Low-Molecular-Weight Hyaluronic Acid by Near-Infrared Spectroscopy: A Comparison between Traditional Machine Learning and Deep Learning

Confusing low-molecular-weight hyaluronic acid (LMWHA) from acid degradation and enzymatic hydrolysis (named LMWHA-A and LMWHA-E, respectively) will lead to health hazards and commercial risks. The purpose of this work is to analyze the structural differences between LMWHA-A and LMWHA-E, and then achieve a fast and accurate classification based on near-infrared (NIR) spectroscopy and machine learning. First, we combined nuclear magnetic resonance (NMR), Fourier transform infrared (FTIR) spectroscopy, two-dimensional correlated NIR spectroscopy (2DCOS), and aquaphotomics to analyze the structural differences between LMWHA-A and LMWHA-E. Second, we compared the dimensionality reduction methods including principal component analysis (PCA), kernel PCA (KPCA), and t-distributed stochastic neighbor embedding (t-SNE). Finally, the differences in classification effect of traditional machine learning methods including partial least squares-discriminant analysis (PLS-DA), support vector classification (SVC), and random forest (RF) as well as deep learning methods including one-dimensional convolutional neural network (1D-CNN) and long short-term memory (LSTM) were compared. The results showed that genetic algorithm (GA)-SVC and RF were the best performers in traditional machine learning, but their highest accuracy in the test dataset was 90%, while the accuracy of 1D-CNN and LSTM models in the training dataset and test dataset classification was 100%. The results of this study show that compared with traditional machine learning, the deep learning models were better for the classification of LMWHA-A and LMWHA-E. Our research provides a new methodological reference for the rapid and accurate classification of biological macromolecules.

Strategies for the content determination of capsaicin and the identification of adulterated pepper powder using a hand-held near-infrared spectrometer

To achieve the goals of rapid content determination of capsaicin and adulteration detection of pepper powder. The method based on the hand-held near-infrared spectrometer combined with ensemble preprocessing was proposed. DoE-based ensemble preprocessing technique was utilized to develop the partial least squares regression models of red pepper [Capsicum annuum L. var. conoides (Mill.) Irish] powders. The performance of final models was evaluated using coefficient of determination (R²), root mean square error of prediction (RMSEP) and residual predictive deviation (RPD). Model development using selective ensemble preprocessing gave the best prediction of capsaicin in Yanjiao pepper powder (R² = 0.9800, RPD = 7.090, RMSEP = 0.00689) and Tianying pepper powder (R² = 0.8935, RPD = 3.017, RMSEP = 0.06154). Moreover, the potential of grey wolf optimizer-support vector machine (GWO-SVM) to detect adulterated pepper powder was investigated. The samples were composed of two authentic products and three different adulterants with different adulteration levels. The results showed that the classification accuracy of GWO-SVM model for Yanjiao peppers was over 90 %, which realized the adulteration detection of Yanjiao pepper. And GWO-SVM showed better performance in detecting adulterated Tianying pepper compared to hierarchical cluster analysis, orthogonal partial least squares discriminant analysis and random forest. In summary, the quality control strategy established in this paper can provide a solution for the adulteration detection and quality evaluation of pepper powder in a rapid and on-site way.

Using Fuzzy Logic to Increase Accuracy in Mango Maturity Index Classification: Approach for Developing a Portable Near-Infrared Spectroscopy Device

Grading is a decisive step in the successful distribution of mangoes to customers according to their preferences for the maturity index. A non-destructive method using near-infrared spectroscopy has historically been used to predict the maturity of fruit. This research classifies the maturity indexes in five classes using a new approach involving classification modeling and the application of fuzzy logic and indirect classification by measuring four parameters: total acidity, soluble solids content, firmness, and starch. These four quantitative parameters provide guidelines for maturity indexes and consumer preferences. The development of portable devices uses a neo spectra micro development kit with specifications for the spectrum of 1350-2500 nm. In terms of computer technology, this study uses a Raspberry Pi and Python programming. To improve the accuracy performance, preprocessing is carried out using 12 spectral transformation operators. Next, these operators are collected and combined to achieve optimal performance. The performance of the classification model with direct and indirect approaches is then compared. Ultimately, classification of the direct approach with preprocessing using linear discriminant analysis offered an accuracy of 91.43%, and classification of the indirect approach using partial least squares with fuzzy logic had an accuracy of 95.7%.

Near-Infrared Spectroscopy Enables Arthroscopic Histologic Grading of Human Knee Articular Cartilage

Purpose

To develop the means to estimate cartilage histologic grades and proteoglycan content in ex vivo arthroscopy using near-infrared spectroscopy (NIRS).

Methods

In this experimental study, arthroscopic NIR spectral measurements were performed on both knees of 9 human cadavers, followed by osteochondral block extraction and in vitro measurements: reacquisition of spectra and reference measurements (proteoglycan content, and three histologic scores). A hybrid model, combining principal component analysis and linear mixed-effects model (PCA-LME), was trained for each reference to investigate its relationship with in vitro NIR spectra. The performance of the PCA-LME model was validated with ex vivo spectra before and after the exclusion of outlying spectra. Model performance was evaluated based on Spearman rank correlation (ρ) and root-mean-square error (RMSE).

Results

The PCA-LME models performed well (independent test: average ρ = 0.668, RMSE = 0.892, P < .001) in the prediction of the reference measurements based on in vitro data. The performance on ex vivo arthroscopic data was poorer but improved substantially after outlier exclusion (independent test: average ρ = 0.462 to 0.614, RMSE = 1.078 to 0.950, P = .019 to .008).

Conclusions

NIRS is capable of nondestructive evaluation of cartilage integrity (i.e., histologic scores and proteoglycan content) under similar conditions as in clinical arthroscopy.

Clinical Relevance

There are clear clinical benefits to the accurate assessment of cartilage lesions in arthroscopy. Visual grading is the current standard of care. However, optical techniques, such as NIRS, may provide a more objective assessment of cartilage damage.

Discrimination of Substandard and Falsified Formulations from Genuine Pharmaceuticals Using NIR Spectra and Machine Learning

Near-infrared (NIR) spectroscopy is a promising technique for field identification of substandard and falsified drugs because it is portable, rapid, nondestructive, and can differentiate many formulated pharmaceutical products. Portable NIR spectrometers rely heavily on chemometric analyses based on libraries of NIR spectra from authentic pharmaceutical samples. However, it is difficult to build comprehensive product libraries in many low- and middle-income countries due to the large numbers of manufacturers who supply these markets, frequent unreported changes in materials sourcing and product formulation by the manufacturers, and general lack of cooperation in providing authentic samples. In this work, we show that a simple library of lab-formulated binary mixtures of an active pharmaceutical ingredient (API) with two diluents gave good performance on field screening tasks, such as discriminating substandard and falsified formulations of the API. Six data analysis models, including principal component analysis and support-vector machine classification and regression methods and convolutional neural networks, were trained on binary mixtures of acetaminophen with either lactose or ascorbic acid. While the models all performed strongly in cross-validation (on formulations similar to their training set), they individually showed poor robustness for formulations outside the training set. However, a predictive algorithm based on the six models, trained only on binary samples, accurately predicts whether the correct amount of acetaminophen is present in ternary mixtures, genuine acetaminophen formulations, adulterated acetaminophen formulations, and falsified formulations containing substitute APIs. This data analytics approach may extend the utility of NIR spectrometers for analysis of pharmaceuticals in low-resource settings.

Near infrared spectroscopic evaluation of biochemical and crimp properties of knee joint ligaments and patellar tendon

Knee ligaments and tendons play an important role in stabilizing and controlling the motions of the knee. Injuries to the ligaments can lead to abnormal mechanical loading of the other supporting tissues (e.g., cartilage and meniscus) and even osteoarthritis. While the condition of knee ligaments can be examined during arthroscopic repair procedures, the arthroscopic evaluation suffers from subjectivity and poor repeatability. Near infrared spectroscopy (NIRS) is capable of non-destructively quantifying the composition and structure of collagen-rich connective tissues, such as articular cartilage and meniscus. Despite the similarities, NIRS-based evaluation of ligament composition has not been previously attempted. In this study, ligaments and patellar tendon of ten bovine stifle joints were measured with NIRS, followed by chemical and histological reference analysis. The relationship between the reference properties of the tissue and NIR spectra was investigated using partial least squares regression. NIRS was found to be sensitive towards the water (R2_CV = .65) and collagen (R2_CV = .57) contents, while elastin, proteoglycans, and the internal crimp structure remained undetectable. As collagen largely determines the mechanical response of ligaments, we conclude that NIRS demonstrates potential for quantitative evaluation of knee ligaments.

Evaluating the impact of NIR pre-processing methods via multiblock partial least-squares

Near-infrared (NIR) spectral data are used in many applications to predict physical and chemical properties. However, it can result in poor predictive models when untreated spectra are directly used to estimate these properties. Many pre-preprocessing techniques are available to reduce noise and variance unrelated to the studied property but choosing which one to apply can be tricky. Existing methods to select a pre-processing are time-consuming or do not allow for a meaningful comparison of the different techniques. Even though new methods focus on extracting complementary information from each pre-processing, an optimal combination is still required to obtain efficient predictive models and avoid extensive computational costs. Here, we propose an approach using multiblock partial least squares (MBPLS) to simultaneously compare the impact of the pre-processing techniques on spectral data and as a result on the regression models. Superloadings provide qualitative and quantitative information on pre-processed data. This tool helps compare and determine which pre-processing technique, or combinations thereof, that may be appropriate for a dataset, not just a single "best" one. Using this, the analyst is then better equipped to make a final choice when selecting which ones to include. This method is tested on artificial signals and NIR spectra from corn samples.

Trends in artificial intelligence, machine learning, and chemometrics applied to chemical data

Artificial intelligence-based methods such as chemometrics, machine learning, and deep learning are promising tools that lead to a clearer and better understanding of data. Only with these tools, data can be used to its full extent, and the gained knowledge on processes, interactions, and characteristics of the sample is maximized. Therefore, scientists are developing data science tools mentioned above to automatically and accurately extract information from data and increase the application possibilities of the respective data in various fields. Accordingly, AI-based techniques were utilized for chemical data since the 1970s and this review paper focuses on the recent trends of chemometrics, machine learning, and deep learning for chemical and spectroscopic data in 2020. In this regard, inverse modeling, preprocessing methods, and data modeling applied to spectra and image data for various measurement techniques are discussed.

Characterization of connective tissues using near-infrared spectroscopy and imaging

Near-infrared (NIR) spectroscopy is a powerful analytical method for rapid, non-destructive and label-free assessment of biological materials. Compared to mid-infrared spectroscopy, NIR spectroscopy excels in penetration depth, allowing intact biological tissue assessment, albeit at the cost of reduced molecular specificity. Furthermore, it is relatively safe compared to Raman spectroscopy, with no risk of laser-induced photothermal damage. A typical NIR spectroscopy workflow for biological tissue characterization involves sample preparation, spectral acquisition, pre-processing and analysis. The resulting spectrum embeds intrinsic information on the tissue's biomolecular, structural and functional properties. Here we demonstrate the analytical power of NIR spectroscopy for exploratory and diagnostic applications by providing instructions for acquiring NIR spectra, maps and images in biological tissues. By adapting and extending this protocol from the demonstrated application in connective tissues to other biological tissues, we expect that a typical NIR spectroscopic study can be performed by a non-specialist user to characterize biological tissues in basic research or clinical settings. We also describe how to use this protocol for exploratory study on connective tissues, including differentiating among ligament types, non-destructively monitoring changes in matrix formation during engineered cartilage development, mapping articular cartilage proteoglycan content across bovine patella and spectral imaging across the depth-wise zones of articular cartilage and subchondral bone. Depending on acquisition mode and experiment objectives, a typical exploratory study can be completed within 6 h, including sample preparation and data analysis.

Improved prediction of fuel properties with near-infrared spectroscopy using a complementary sequential fusion of scatter correction techniques

Near-infrared (NIR) spectroscopy of fuels can suffer from scattering effects which may mask the signals corresponding to key analytes in the spectra. Therefore, scatter correction techniques are often used prior to any modelling so to remove scattering and improve predictive performances. However, different scatter correction techniques may carry complementary information so that, if jointly used, both model stability and performances could be improved. A solution to that is the fusion of complementary information from differently scatter corrected data. In the present work, the use of a preprocessing fusion approach called sequential preprocessing through orthogonalization (SPORT) is demonstrated for predicting key quality parameters in diesel fuels. In particular, the possibility of predicting four different key properties, i.e., boiling point (°C), density (g/mL), aromatic mass (%) and viscosity (cSt), was considered. As a comparison, standard partial least-squares (PLS) regression modelling was performed on data pretreated by SNV and 2nd derivative (which is a widely used preprocessing combination). The results showed that the SPORT models, based on the fusion of scatter correction techniques, outperformed the standard PLS models in the prediction of all the four properties, suggesting that selection and use of a single scatter correction technique is often not sufficient. Up to complete bias removal with 50% reduction in prediction error was obtained. The R²_P was increased by up to 8%. The sequential scatter fusion approach (SPORT) is not limited to NIR data but can be applied to any other spectral data where a preprocessing optimization step is required.