Gaussian Process Functional Regression Modeling for Batch Data

Bayesian Regression Analysis for Dependent Data with an Elliptical Shape

This paper proposes a parametric hierarchical model for functional data with an elliptical shape, using a Gaussian process prior to capturing the data dependencies that reflect systematic errors while modeling the underlying curved shape through a von Mises-Fisher distribution. The model definition, Bayesian inference, and MCMC algorithm are discussed. The effectiveness of the model is demonstrated through the reconstruction of curved trajectories using both simulated and real-world examples. The discussion in this paper focuses on two-dimensional problems, but the framework can be extended to higher-dimensional spaces, making it adaptable to a wide range of applications.

Predicting biochemical oxygen demand in wastewater treatment plant using advance extreme learning machine optimized by Bat algorithm

Wastewater quality modelling plays a vital role in planning and management of wastewater treatment plants (WWTP). This paper develops a new hybrid machine learning model based on extreme learning machine (ELM) optimized by Bat algorithm (ELM-Bat) for modelling five day effluent biochemical oxygen demand (BOD5). Specifically, this hybrid model combines the Bat algorithm for model parameters optimization and the standalone ELM. The proposed model was developed using historical measured effluents wastewater quality variables, i.e., the chemical oxygen demand (COD), temperature, pH, total suspended solid (TSS), specific conductance (SC) and the wastewater flow (Q). The performances of the hybrid ELM-Bat were compared with those of the multilayer perceptron neural network (MLPNN), the random forest regression (RFR), the Gaussian process regression (GPR), the random vector functional link network (RVFL), and the multiple linear regression (MLR) models. By comparing several input variables combination, the improvement achieved in the accuracy of prediction through the hybrid ELM-Bat was quantified. All models were first calibrated using training dataset and later tested using validation and based on four performances metrics namely, root mean square error (RMSE), mean absolute error (MAE), the correlation coefficient (R), and the Nash-Sutcliffe model efficiency (NSE). In all, it is concluded that the ELM-Bat is the most accurate model when all the six input were included as input variables, and it outperforms all other benchmark models in terms of predictive accuracy, exhibiting RMSE, MAE, R and NSE values of approximately, 0.885, 0.781, 2.621, and 1.989, respectively.

Permutation-Based Inference for Function-on-Scalar Regression With an Application in PET Brain Imaging

The density of various proteins throughout the human brain can be studied through the use of positron emission tomography (PET) imaging. We report here on data from a study of serotonin transporter (5-HTT) binding. While PET imaging data analysis is most commonly performed on data that are aggregated into several discrete a priori regions of interest, in this study, primary interest is on measures of 5-HTT binding potential that are made at many locations along a continuous anatomically defined tract, one that was chosen to follow serotonergic axons. Our goal is to characterize the binding patterns along this tract and also to determine how such patterns differ between control subjects and depressed patients. Due to the nature of our data, we utilize function-on-scalar regression modeling to make optimal use of our data. Inference on both main effects (position along the tract; diagnostic group) and their interactions is made using permutation testing strategies that do not require distributional assumptions. Also, to investigate the question of homogeneity we implement a permutation testing strategy, which adapts a "block bootstrapping" approach from time series analysis to the functional data setting.

MAGMA: inference and prediction using multi-task Gaussian processes with common mean

A novel multi-task Gaussian process (GP) framework is proposed, by using a common mean process for sharing information across tasks. In particular, we investigate the problem of time series forecasting, with the objective to improve multiple-step-ahead predictions. The common mean process is defined as a GP for which the hyper-posterior distribution is tractable. Therefore an EM algorithm is derived for handling both hyper-parameters optimisation and hyper-posterior computation. Unlike previous approaches in the literature, the model fully accounts for uncertainty and can handle irregular grids of observations while maintaining explicit formulations, by modelling the mean process in a unified GP framework. Predictive analytical equations are provided, integrating information shared across tasks through a relevant prior mean. This approach greatly improves the predictive performances, even far from observations, and may reduce significantly the computational complexity compared to traditional multi-task GP models. Our overall algorithm is called Magma (standing for Multi tAsk GPs with common MeAn). The quality of the mean process estimation, predictive performances, and comparisons to alternatives are assessed in various simulated scenarios and on real datasets.

Robust Non-Parametric Mortality and Fertility Modelling and Forecasting: Gaussian Process Regression Approaches

A rapid decline in mortality and fertility has become major issues in many developed countries over the past few decades. An accurate model for forecasting demographic movements is important for decision making in social welfare policies and resource budgeting among the government and many industry sectors. This article introduces a novel non-parametric approach using Gaussian process regression with a natural cubic spline mean function and a spectral mixture covariance function for mortality and fertility modelling and forecasting. Unlike most of the existing approaches in demographic modelling literature, which rely on time parameters to determine the movements of the whole mortality or fertility curve shifting from one year to another over time, we consider the mortality and fertility curves from their components of all age-specific mortality and fertility rates and assume each of them following a Gaussian process over time to fit the whole curves in a discrete but intensive style. The proposed Gaussian process regression approach shows significant improvements in terms of forecast accuracy and robustness compared to other mainstream demographic modelling approaches in the short-, mid- and long-term forecasting using the mortality and fertility data of several developed countries in the numerical examples.

An Experimental and Statistical Study on Rebar Corrosion Considering the Temperature Effect Using Gaussian Process Regression

Temperature is an important factor that affects corrosion potential in rebars. The temperature effect must be removed from the corrosion potential for precise measurement of corrosion rates. To separate the temperature effect from the corrosion potential, in this study rebar specimens were not embedded in concrete but, instead, were placed in an uncontrolled air environment. Gaussian process regression (GPR) was applied to the temperature and the non-corrosion potential data in order to remove the temperature effect from the corrosion potential. The results indicated that the corrosion potential was affected by the temperature. Furthermore, the GPR models of all the experimental cases showed high coefficients of determination (R2 > 0.90) and low root mean square errors (RMSE < 0.08), meaning that these models had high reliability. The fitted GPR models were used to successfully remove the temperature effect from the corrosion potential. This demonstrates that the GPR method can be appropriately used to assess the temperature effect on rebar corrosion.

SOH and RUL Prediction of Lithium-Ion Batteries Based on Gaussian Process Regression with Indirect Health Indicators

The state of health (SOH) and remaining useful life (RUL) of lithium-ion batteries are two important factors which are normally predicted using the battery capacity. However, it is difficult to directly measure the capacity of lithium-ion batteries for online applications. In this paper, indirect health indicators (IHIs) are extracted from the curves of voltage, current, and temperature in the process of charging and discharging lithium-ion batteries, which respond to the battery capacity degradation process. A few reasonable indicators are selected as the inputs of SOH prediction by the grey relation analysis method. The short-term SOH prediction is carried out by combining the Gaussian process regression (GPR) method with probability predictions. Then, considering that there is a certain mapping relationship between SOH and RUL, three IHIs and the present SOH value are utilized to predict RUL of lithium-ion batteries through the GPR model. The results show that the proposed method has high prediction accuracy.

Robust functional regression model for marginal mean and subject-specific inferences

We introduce flexible robust functional regression models, using various heavy-tailed processes, including a Student t-process. We propose efficient algorithms in estimating parameters for the marginal mean inferences and in predicting conditional means as well as interpolation and extrapolation for the subject-specific inferences. We develop bootstrap prediction intervals (PIs) for conditional mean curves. Numerical studies show that the proposed model provides a robust approach against data contamination or distribution misspecification, and the proposed PIs maintain the nominal confidence levels. A real data application is presented as an illustrative example.

A Two-Layer Mixture Model of Gaussian Process Functional Regressions and Its MCMC EM Algorithm

The mixture of Gaussian processes (GPs) is capable of learning any general stochastic process based on a given set of (sample) curves for the regression and prediction problems. However, it is ineffective for curve clustering and prediction, when the sample curves are derived from different stochastic processes as independent sources linearly mixed together. In this paper, we propose a two-layer mixture model of GP functional regressions (GPFRs) to describe such a mixture of general stochastic processes or independent sources, especially for curve clustering and prediction. Specifically, in the lower layer, the mixture of GPFRs (MGPFRs) is developed for a cluster (or class) of curves within the input space. In the higher layer, the mixture of MGPFRs is further established to divide the curves into clusters according to its components in the output space. For the parameter estimation of the two-layer mixture of GPFRs, we develop a Monte Carlo EM algorithm based on a Monte Carlo Markov chain (MCMC) method, in short, the MCMC EM algorithm. We validate the hierarchical mixture of GPFRs and MCMC EM algorithm using synthetic and real-world data sets. Our results show that our new model outperforms the conventional mixture models in curve clustering and prediction.

Efficient Bayesian hierarchical functional data analysis with basis function approximations using Gaussian-Wishart processes

Functional data are defined as realizations of random functions (mostly smooth functions) varying over a continuum, which are usually collected on discretized grids with measurement errors. In order to accurately smooth noisy functional observations and deal with the issue of high-dimensional observation grids, we propose a novel Bayesian method based on the Bayesian hierarchical model with a Gaussian-Wishart process prior and basis function representations. We first derive an induced model for the basis-function coefficients of the functional data, and then use this model to conduct posterior inference through Markov chain Monte Carlo methods. Compared to the standard Bayesian inference that suffers serious computational burden and instability in analyzing high-dimensional functional data, our method greatly improves the computational scalability and stability, while inheriting the advantage of simultaneously smoothing raw observations and estimating the mean-covariance functions in a nonparametric way. In addition, our method can naturally handle functional data observed on random or uncommon grids. Simulation and real studies demonstrate that our method produces similar results to those obtainable by the standard Bayesian inference with low-dimensional common grids, while efficiently smoothing and estimating functional data with random and high-dimensional observation grids when the standard Bayesian inference fails. In conclusion, our method can efficiently smooth and estimate high-dimensional functional data, providing one way to resolve the curse of dimensionality for Bayesian functional data analysis with Gaussian-Wishart processes.

A general framework for functional regression modelling

Researchers are increasingly interested in regression models for functional data. This article discusses a comprehensive framework for additive (mixed) models for functional responses and/or functional covariates based on the guiding principle of reframing functional regression in terms of corresponding models for scalar data, allowing the adaptation of a large body of existing methods for these novel tasks. The framework encompasses many existing as well as new models. It includes regression for ‘generalized’ functional data, mean regression, quantile regression as well as generalized additive models for location, shape and scale (GAMLSS) for functional data. It admits many flexible linear, smooth or interaction terms of scalar and functional covariates as well as (functional) random effects and allows flexible choices of bases—particularly splines and functional principal components—and corresponding penalties for each term. It covers functional data observed on common (dense) or curve-specific (sparse) grids. Penalized-likelihood-based and gradient-boosting-based inference for these models are implemented in R packages refund and FDboost , respectively. We also discuss identifiability and computational complexity for the functional regression models covered. A running example on a longitudinal multiple sclerosis imaging study serves to illustrate the flexibility and utility of the proposed model class. Reproducible code for this case study is made available online.

Bayesian Nonparametric Estimation for Dynamic Treatment Regimes with Sequential Transition Times

We analyze a dataset arising from a clinical trial involving multi-stage chemotherapy regimes for acute leukemia. The trial design was a 2 × 2 factorial for frontline therapies only. Motivated by the idea that subsequent salvage treatments affect survival time, we model therapy as a dynamic treatment regime (DTR), that is, an alternating sequence of adaptive treatments or other actions and transition times between disease states. These sequences may vary substantially between patients, depending on how the regime plays out. To evaluate the regimes, mean overall survival time is expressed as a weighted average of the means of all possible sums of successive transitions times. We assume a Bayesian nonparametric survival regression model for each transition time, with a dependent Dirichlet process prior and Gaussian process base measure (DDP-GP). Posterior simulation is implemented by Markov chain Monte Carlo (MCMC) sampling. We provide general guidelines for constructing a prior using empirical Bayes methods. The proposed approach is compared with inverse probability of treatment weighting, including a doubly robust augmented version of this approach, for both single-stage and multi-stage regimes with treatment assignment depending on baseline covariates. The simulations show that the proposed nonparametric Bayesian approach can substantially improve inference compared to existing methods. An R program for implementing the DDP-GP-based Bayesian nonparametric analysis is freely available at https://www.ma.utexas.edu/users/yxu/.

Assessment of DPOAE test-retest difference curves via hierarchical Gaussian processes

Distortion product otoacoustic emissions (DPOAE) testing is a promising alternative to behavioral hearing tests and auditory brainstem response testing of pediatric cancer patients. The central goal of this study is to assess whether significant changes in the DPOAE frequency/emissions curve (DP-gram) occur in pediatric patients in a test-retest scenario. This is accomplished through the construction of normal reference charts, or credible regions, that DP-gram differences lie in, as well as contour probabilities that measure how abnormal (or in a certain sense rare) a test-retest difference is. A challenge is that the data were collected over varying frequencies, at different time points from baseline, and on possibly one or both ears. A hierarchical structural equation Gaussian process model is proposed to handle the different sources of correlation in the emissions measurements, wherein both subject-specific random effects and variance components governing the smoothness and variability of each child's Gaussian process are coupled together.

Estimating Mixture of Gaussian Processes by Kernel Smoothing

When the functional data are not homogeneous, e.g., there exist multiple classes of functional curves in the dataset, traditional estimation methods may fail. In this paper, we propose a new estimation procedure for the Mixture of Gaussian Processes, to incorporate both functional and inhomogeneous properties of the data. Our method can be viewed as a natural extension of high-dimensional normal mixtures. However, the key difference is that smoothed structures are imposed for both the mean and covariance functions. The model is shown to be identifiable, and can be estimated efficiently by a combination of the ideas from EM algorithm, kernel regression, and functional principal component analysis. Our methodology is empirically justified by Monte Carlo simulations and illustrated by an analysis of a supermarket dataset.

Mixed-effects Gaussian process functional regression models with application to dose-response curve prediction

We propose a new semiparametric model for functional regression analysis, combining a parametric mixed-effects model with a nonparametric Gaussian process regression model, namely a mixed-effects Gaussian process functional regression model. The parametric component can provide explanatory information between the response and the covariates, whereas the nonparametric component can add nonlinearity. We can model the mean and covariance structures simultaneously, combining the information borrowed from other subjects with the information collected from each individual subject. We apply the model to dose-response curves that describe changes in the responses of subjects for differing levels of the dose of a drug or agent and have a wide application in many areas. We illustrate the method for the management of renal anaemia. An individual dose-response curve is improved when more information is included by this mechanism from the subject/patient over time, enabling a patient-specific treatment regime.

Penalized gaussian process regression and classification for high-dimensional nonlinear data

The model based on Gaussian process (GP) prior and a kernel covariance function can be used to fit nonlinear data with multidimensional covariates. It has been used as a flexible nonparametric approach for curve fitting, classification, clustering, and other statistical problems, and has been widely applied to deal with complex nonlinear systems in many different areas particularly in machine learning. However, it is a challenging problem when the model is used for the large-scale data sets and high-dimensional data, for example, for the meat data discussed in this article that have 100 highly correlated covariates. For such data, it suffers from large variance of parameter estimation and high predictive errors, and numerically, it suffers from unstable computation. In this article, penalized likelihood framework will be applied to the model based on GPs. Different penalties will be investigated, and their ability in application given to suit the characteristics of GP models will be discussed. The asymptotic properties will also be discussed with the relevant proofs. Several applications to real biomechanical and bioinformatics data sets will be reported.