Opening the kernel of kernel partial least squares and support vector machines

Increased interpretation of deep learning models using hierarchical cluster-based modelling

Linear prediction models based on data with large inhomogeneity or abrupt non-linearities often perform poorly because relationships between groups in the data dominate the model. Given that the data is locally linear, this can be overcome by splitting the data into smaller clusters and creating a local model within each cluster. In this study, the previously published Hierarchical Cluster-based Partial Least Squares Regression (HC-PLSR) procedure was extended to deep learning, in order to increase the interpretability of the deep learning models through local modelling. Hierarchical Cluster-based Convolutional Neural Networks (HC-CNNs), Hierarchical Cluster-based Recurrent Neural Networks (HC-RNNs) and Hierarchical Cluster-based Support Vector Regression models (HC-SVRs) were implemented and tested on spectroscopic data consisting of Fourier Transform Infrared (FT-IR) measurements of raw material dry films, for prediction of average molecular weight during hydrolysis and a simulated data set constructed to contain three clusters of observations with different non-linear relationships between the independent variables and the response. HC-CNN, HC-RNN and HC-SVR outperformed HC-PLSR for the simulated data set, showing the disadvantage of PLSR for highly non-linear data, but for the FT-IR data set there was little to gain in prediction ability from using more complex models than HC-PLSR. Local modelling can ease the interpretation of deep learning models through highlighting differences in feature importance between different regions of the input or output space. Our results showed clear differences between the feature importance for the various local models, which demonstrate the advantages of a local modelling approach with regards to interpretation of deep learning models.

Unsupervised model adaptation for multivariate calibration by domain adaptation-regularization based kernel partial least square

In chemometrics, calibration model adaptation is desired when training- and test-samples come from different distributions. Domain-invariant feature representation is currently a successful strategy to realize model adaptation and has received wide attention. The paper presents a nonlinear unsupervised model adaptation method termed as domain adaption regularization-based kernel partial least squares regression (DarKPLS). DarKPLS aims to minimize the source and target distributions in a low-dimensional latent space projected from the reproducing kernel Hilbert space (RKHS) generated with the labeled source data and unlabeled target data. Specially, the distributional means and variances between source and target latent variables are aligned in the RKHS. By extending existing domain invariant partial least square regression (di-PLS) with the projected maximum mean discrepancy (PMMD) to reduce the mean discrepancy in the RKHS further, DarKPLS could realize fine-grained domain alignment that further improves the adaptation performance. DarKPLS is applied to the γ-polyglutamic acid fermentation dataset, tobacco dataset and corn dataset, and it demonstrates improved prediction results in comparison with No adaptation partial least squares (PLS), null augmented regression (NAR), extended linear joint trained framework (ExtJT), scatter component analysis (SCA) and domain-invariant iterative partial least squares (DIPALS).

The Correlation Analysis between Air Quality and Construction Sites: Evaluation in the Urban Environment during the COVID-19 Pandemic

This research studies the data on air quality and construction activities from 29 January 2020 to 30 April 2020. The analysis focuses on three sample districts of Hangzhou’s Xiacheng, Gongshu, and Xiaoshan districts. The samples, respectively, represent low-level, mid-level, and high-level districts in the scale of construction projects. The correlative relationships are investigated, respectively, in the periods of ‘pandemic lockdown (29 January 2020–20 February 2020)’ and ‘after pandemic lockdown (21 February 2020–30 April 2020)’. The correlative equations are obtained. Based on the guideline values of air parameters provided by the Chinese criteria and standards, the recommended maximum scales of construction projects are defined. The numbers of construction sites are 16, 118, and 311 for the Xiacheng, Gongshu, and Xiaoshan districts during the imposed lockdown period, respectively, and 19, 88, 234, respectively, after the lockdown period. Because the construction site is only one influential factor on the air quality, and the database is not large enough, there are some limitations in the mathematical model and the management plan. Possible problem solving techniques and future studies are introduced at the end of the research study.

Advanced data fusion: Random forest proximities and pseudo-sample principle towards increased prediction accuracy and variable interpretation

Data fusion has gained much attention in the field of life sciences, and this is because analysis of biological samples may require the use of data coming from multiple complementary sources to express the samples fully. Data fusion lies in the idea that different data platforms detect different biological entities. Therefore, if these different biological compounds are then combined, they can provide comprehensive profiling and understanding of the research question in hand. Data fusion can be performed in three different traditional ways: low-level, mid-level, and high-level data fusion. However, the increasing complexity and amount of generated data require the development of more sophisticated fusion approaches. In that regard, the current study presents an advanced data fusion approach (i.e. proximities stacking) based on random forest proximities coupled with the pseudo-sample principle. Four different data platforms of 130 samples each (faecal microbiome, blood, blood headspace, and exhaled breath samples of patients who have Crohn's disease) were used to demonstrate the classification performance of this new approach. More specifically, 104 samples were used to train and validate the models, whereas the remaining 26 samples were used to validate the models externally. Mid-level, high-level, as well as individual platform classification predictions, were made and compared against the proximities stacking approach. The performance of each approach was assessed by calculating the sensitivity and specificity of each model for the external test set, and visualized by performing principal component analysis on the proximity matrices of the training samples to then, subsequently, project the test samples onto that space. The implementation of pseudo-samples allowed for the identification of the most important variables per platform, finding relations among variables of the different data platforms, and the examination of how variables behave in the samples. The proximities stacking approach outperforms both mid-level and high-level fusion approaches, as well as all individual platform predictions. Concurrently, it tackles significant bottlenecks of the traditional ways of fusion and of another advanced fusion way discussed in the paper, and finally, it contradicts the general belief that the more data, the merrier the result, and therefore, considerations have to be taken into account before any data fusion analysis is conducted.

Application of machine learning methods for the prediction of organic solid waste treatment and recycling processes: A review

Conventional treatment and recycling methods of organic solid waste contain inherent flaws, such as low efficiency, low accuracy, high cost, and potential environmental risks. In the past decade, machine learning has gradually attracted increasing attention in solving the complex problems of organic solid waste treatment. Although significant research has been carried out, there is a lack of a systematic review of the research findings in this field. This study sorts the research studies published between 2003 and 2020, summarizes the specific application fields, characteristics, and suitability of different machine learning models, and discusses the relevant application limitations and future prospects. It can be concluded that studies mostly focused on municipal solid waste management, followed by anaerobic digestion, thermal treatment, composting, and landfill. The most widely used model is the artificial neural network, which has been successfully applied to various complicated non-linear organic solid waste related problems.

Chemometric Strategies for Spectroscopy-Based Food Authentication

In the last decades, spectroscopic techniques have played an increasingly crucial role in analytical chemistry, due to the numerous advantages they offer. Several of these techniques (e.g., Near-InfraRed—NIR—or Fourier Transform InfraRed—FT-IR—spectroscopy) are considered particularly valuable because, by means of suitable equipment, they enable a fast and non-destructive sample characterization. This aspect, together with the possibility of easily developing devices for on- and in-line applications, has recently favored the diffusion of such approaches especially in the context of foodstuff quality control. Nevertheless, the complex nature of the signal yielded by spectroscopy instrumentation (regardless of the spectral range investigated) inevitably calls for the use of multivariate chemometric strategies for its accurate assessment and interpretation. This review aims at providing a comprehensive overview of some of the chemometric tools most commonly exploited for spectroscopy-based foodstuff analysis and authentication. More in detail, three different scenarios will be surveyed here: data exploration, calibration and classification. The main methodologies suited to addressing each one of these different tasks will be outlined and examples illustrating their use will be provided alongside their description.

Constructing bi-plots for random forest: Tutorial

Current technological developments have allowed for a significant increase and availability of data. Consequently, this has opened enormous opportunities for the machine learning and data science field, translating into the development of new algorithms in a wide range of applications in medical, biomedical, daily-life, and national security areas. Ensemble techniques are among the pillars of the machine learning field, and they can be defined as approaches in which multiple, complex, independent/uncorrelated, predictive models are subsequently combined by either averaging or voting to yield a higher model performance. Random forest (RF), a popular ensemble method, has been successfully applied in various domains due to its ability to build predictive models with high certainty and little necessity of model optimization. RF provides both a predictive model and an estimation of the variable importance. However, the estimation of the variable importance is based on thousands of trees, and therefore, it does not specify which variable is important for which sample group. The present study demonstrates an approach based on the pseudo-sample principle that allows for construction of bi-plots (i.e. spin plots) associated with RF models. The pseudo-sample principle for RF. is explained and demonstrated by using two simulated datasets, and three different types of real data, which include political sciences, food chemistry and the human microbiome data. The pseudo-sample bi-plots, associated with RF and its unsupervised version, allow for a versatile visualization of multivariate models, and the variable importance and the relation among them.

Decision Tree-PLS (DT-PLS) algorithm for the development of process: Specific local prediction models

This work presents a novel multivariate statistical algorithm, Decision Tree-PLS (DT-PLS), to improve the prediction and understanding of dynamic processes based on local partial least square regression (PLSR) models for characteristic process groups defined based on Decision Tree (DT) analysis. The DT-PLS algorithm is successfully applied to two different cell culture data sets, one obtained from bioreactors of 3.5 L lab scale and the other obtained from the 15 ml ambr microbioreactor system. Substantial improvement in the predictive capabilities of the model can be achieved based on the localization compared to the classical PLSR approach, which is implemented in the commercially available packages. Additionally, the differences in the model parameters of the local models suggest that the governing process variables vary for the different process regimes indicating the different states of the cell under different process conditions.

SVM-RFE: selection and visualization of the most relevant features through non-linear kernels

Background

Support vector machines (SVM) are a powerful tool to analyze data with a number of predictors approximately equal or larger than the number of observations. However, originally, application of SVM to analyze biomedical data was limited because SVM was not designed to evaluate importance of predictor variables. Creating predictor models based on only the most relevant variables is essential in biomedical research. Currently, substantial work has been done to allow assessment of variable importance in SVM models but this work has focused on SVM implemented with linear kernels. The power of SVM as a prediction model is associated with the flexibility generated by use of non-linear kernels. Moreover, SVM has been extended to model survival outcomes. This paper extends the Recursive Feature Elimination (RFE) algorithm by proposing three approaches to rank variables based on non-linear SVM and SVM for survival analysis.

Results

The proposed algorithms allows visualization of each one the RFE iterations, and hence, identification of the most relevant predictors of the response variable. Using simulation studies based on time-to-event outcomes and three real datasets, we evaluate the three methods, based on pseudo-samples and kernel principal component analysis, and compare them with the original SVM-RFE algorithm for non-linear kernels. The three algorithms we proposed performed generally better than the gold standard RFE for non-linear kernels, when comparing the truly most relevant variables with the variable ranks produced by each algorithm in simulation studies. Generally, the RFE-pseudo-samples outperformed the other three methods, even when variables were assumed to be correlated in all tested scenarios.

Conclusions

The proposed approaches can be implemented with accuracy to select variables and assess direction and strength of associations in analysis of biomedical data using SVM for categorical or time-to-event responses. Conducting variable selection and interpreting direction and strength of associations between predictors and outcomes with the proposed approaches, particularly with the RFE-pseudo-samples approach can be implemented with accuracy when analyzing biomedical data. These approaches, perform better than the classical RFE of Guyon for realistic scenarios about the structure of biomedical data.

Electronic supplementary material

The online version of this article (10.1186/s12859-018-2451-4) contains supplementary material, which is available to authorized users.

Batch process fault detection and identification based on discriminant global preserving kernel slow feature analysis

As an attractive nonlinear dynamic data analysis tool, global preserving kernel slow feature analysis (GKSFA) has achieved great success in extracting the high nonlinearity and inherently time-varying dynamics of batch process. However, GKSFA is an unsupervised feature extraction method and lacks the ability to utilize batch process class label information, which may not offer the most effective means for dealing with batch process monitoring. To overcome this problem, we propose a novel batch process monitoring method based on the modified GKSFA, referred to as discriminant global preserving kernel slow feature analysis (DGKSFA), by closely integrating discriminant analysis and GKSFA. The proposed DGKSFA method can extract discriminant feature of batch process as well as preserve global and local geometrical structure information of observed data. For the purpose of fault detection, a monitoring statistic is constructed based on the distance between the optimal kernel feature vectors of test data and normal data. To tackle the challenging issue of nonlinear fault variable identification, a new nonlinear contribution plot method is also developed to help identifying the fault variable after a fault is detected, which is derived from the idea of variable pseudo-sample trajectory projection in DGKSFA nonlinear biplot. Simulation results conducted on a numerical nonlinear dynamic system and the benchmark fed-batch penicillin fermentation process demonstrate that the proposed process monitoring and fault diagnosis approach can effectively detect fault and distinguish fault variables from normal variables.

Differentiation Between Organic and Non-Organic Apples Using Diffraction Grating and Image Processing-A Cost-Effective Approach

As the expectation for higher quality of life increases, consumers have higher demands for quality food. Food authentication is the technical means of ensuring food is what it says it is. A popular approach to food authentication is based on spectroscopy, which has been widely used for identifying and quantifying the chemical components of an object. This approach is non-destructive and effective but expensive. This paper presents a computer vision-based sensor system for food authentication, i.e., differentiating organic from non-organic apples. This sensor system consists of low-cost hardware and pattern recognition software. We use a flashlight to illuminate apples and capture their images through a diffraction grating. These diffraction images are then converted into a data matrix for classification by pattern recognition algorithms, including k-nearest neighbors (k-NN), support vector machine (SVM) and three partial least squares discriminant analysis (PLS-DA)- based methods. We carry out experiments on a reasonable collection of apple samples and employ a proper pre-processing, resulting in a highest classification accuracy of 94%. Our studies conclude that this sensor system has the potential to provide a viable solution to empower consumers in food authentication.

Predicting dissolved oxygen concentration using kernel regression modeling approaches with nonlinear hydro-chemical data

Kernel function-based regression models were constructed and applied to a nonlinear hydro-chemical dataset pertaining to surface water for predicting the dissolved oxygen levels. Initial features were selected using nonlinear approach. Nonlinearity in the data was tested using BDS statistics, which revealed the data with nonlinear structure. Kernel ridge regression, kernel principal component regression, kernel partial least squares regression, and support vector regression models were developed using the Gaussian kernel function and their generalization and predictive abilities were compared in terms of several statistical parameters. Model parameters were optimized using the cross-validation procedure. The proposed kernel regression methods successfully captured the nonlinear features of the original data by transforming it to a high dimensional feature space using the kernel function. Performance of all the kernel-based modeling methods used here were comparable both in terms of predictive and generalization abilities. Values of the performance criteria parameters suggested for the adequacy of the constructed models to fit the nonlinear data and their good predictive capabilities.

A quantitative structure-activity relationship study of anti-HIV activity of substituted HEPT using nonlinear models

We performed studies on extended series of 79 HEPT ligands (1-[(2-hydroxyethoxy)methyl]-6-(phenylthio)thymine), inhibitors of HIV reverse-transcriptase with anti-HIV biological activity, using quantitative structure–activity relationship (QSAR) methods that imply analysis of correlations and representation of models. A suitable set of molecular descriptors was calculated, and the genetic algorithm was employed to select those descriptors which resulted in the best-fit models. The kernel partial least square and Levenberg–Marquardt artificial neural network were utilized to construct the nonlinear QSAR models. The proposed methods will be of great significance in this research, and would be expected to apply to other similar research fields.

Linear and non-linear chemometric modeling of THM formation in Barcelona's water treatment plant

The complex behavior observed for the dependence of trihalomethane formation on forty one water treatment plant (WTP) operational variables is investigated by means of linear and non-linear regression methods, including kernel-partial least squares (K-PLS), and support vector machine regression (SVR). Lower prediction errors of total trihalomethane concentrations (lower than 14% for external validation samples) were obtained when these two methods were applied in comparison to when linear regression methods were applied. A new visualization technique revealed the complex nonlinear relationships among the operational variables and displayed the existing correlations between input variables and the kernel matrix on one side and the support vectors on the other side. Whereas some water treatment plant variables like river water TOC and chloride concentrations, and breakpoint chlorination were not considered to be significant due to the multi-collinear effect in straight linear regression modeling methods, they were now confirmed to be significant using K-PLS and SVR non-linear modeling regression methods, proving the better performance of these methods for the prediction of complex formation of trihalomethanes in water disinfection plants.

Interpretation and visualization of non-linear data fusion in kernel space: study on metabolomic characterization of progression of multiple sclerosis

Background

In the last decade data fusion has become widespread in the field of metabolomics. Linear data fusion is performed most commonly. However, many data display non-linear parameter dependences. The linear methods are bound to fail in such situations. We used proton Nuclear Magnetic Resonance and Gas Chromatography-Mass Spectrometry, two well established techniques, to generate metabolic profiles of Cerebrospinal fluid of Multiple Sclerosis (MScl) individuals. These datasets represent non-linearly separable groups. Thus, to extract relevant information and to combine them a special framework for data fusion is required.

Methodology

The main aim is to demonstrate a novel approach for data fusion for classification; the approach is applied to metabolomics datasets coming from patients suffering from MScl at a different stage of the disease. The approach involves data fusion in kernel space and consists of four main steps. The first one is to extract the significant information per data source using Support Vector Machine Recursive Feature Elimination. This method allows one to select a set of relevant variables. In the next step the optimized kernel matrices are merged by linear combination. In step 3 the merged datasets are analyzed with a classification technique, namely Kernel Partial Least Square Discriminant Analysis. In the final step, the variables in kernel space are visualized and their significance established.

Conclusions

We find that fusion in kernel space allows for efficient and reliable discrimination of classes (MScl and early stage). This data fusion approach achieves better class prediction accuracy than analysis of individual datasets and the commonly used mid-level fusion. The prediction accuracy on an independent test set (8 samples) reaches 100%. Additionally, the classification model obtained on fused kernels is simpler in terms of complexity, i.e. just one latent variable was sufficient. Finally, visualization of variables importance in kernel space was achieved.

Screening for emphysema via exhaled volatile organic compounds

Chronic obstructive pulmonary disease (COPD)/emphysema risk groups are well defined and screening allows for early identification of disease. The capability of exhaled volatile organic compounds (VOCs) to detect emphysema, as found by computed tomography (CT) in current and former heavy smokers participating in a lung cancer screening trial, was investigated. CT scans, pulmonary function tests and breath sample collections were obtained from 204 subjects. Breath samples were analyzed with a proton-transfer reaction mass spectrometer (PTR-MS) to obtain VOC profiles listed as ions at various mass-to-charge ratios (m/z). Using bootstrapped stepwise forward logistic regression, we identified specific breath profiles as a potential tool for the diagnosis of emphysema, of airflow limitation or gas-exchange impairment. A marker for emphysema was found at m/z 87 (tentatively attributed to 2-methylbutanal). The area under the receiver operating characteristic curve (ROC) of this marker to diagnose emphysema was 0.588 (95% CI 0.453-0.662). Mass-to-charge ratios m/z 52 (most likely chloramine) and m/z 135 (alkyl benzene) were linked to obstructive disease and m/z 122 (most probably alkyl homologs) to an impaired diffusion capacity. ROC areas were 0.646 (95% CI 0.562-0.730) and 0.671 (95% CI 0.524-0.710), respectively. In the screening setting, exhaled VOCs measured by PTR-MS constitute weak markers for emphysema, pulmonary obstruction and impaired diffusion capacity.