Hybrid Adaptive Splines

Greedy knot selection algorithm for restricted cubic spline regression

Non-linear regression modeling is common in epidemiology for prediction purposes or estimating relationships between predictor and response variables. Restricted cubic spline (RCS) regression is one such method, for example, highly relevant to Cox proportional hazard regression model analysis. RCS regression uses third-order polynomials joined at knot points to model non-linear relationships. The standard approach is to place knots by a regular sequence of quantiles between the outer boundaries. A regression curve can easily be fitted to the sample using a relatively high number of knots. The problem is then overfitting, where a regression model has a good fit to the given sample but does not generalize well to other samples. A low knot count is thus preferred. However, the standard knot selection process can lead to underperformance in the sparser regions of the predictor variable, especially when using a low number of knots. It can also lead to overfitting in the denser regions. We present a simple greedy search algorithm using a backward method for knot selection that shows reduced prediction error and Bayesian information criterion scores compared to the standard knot selection process in simulation experiments. We have implemented the algorithm as part of an open-source R-package, knutar.

Aggregated functional data model applied on clustering and disaggregation of UK electrical load profiles

Understanding electrical energy demand at the consumer level plays an important role in planning the distribution of electrical networks and offering of off-peak tariffs, but observing individual consumption patterns is still expensive. On the other hand, aggregated load curves are normally available at the substation level. The proposed methodology separates substation aggregated loads into estimated mean consumption curves, called typical curves, including information given by explanatory variables. In addition, a model-based clustering approach for substations is proposed based on the similarity of their consumers’ typical curves and covariance structures. The methodology is applied to a real substation load monitoring dataset from the UK and tested in eight simulated scenarios.

Smoothing splines approximation using Hilbert curve basis selection

Smoothing splines have been used pervasively in nonparametric regressions. However, the computational burden of smoothing splines is significant when the sample size n is large. When the number of predictors d ≥ 2 , the computational cost for smoothing splines is at the order of O(n 3) using the standard approach. Many methods have been developed to approximate smoothing spline estimators by using q basis functions instead of n ones, resulting in a computational cost of the order O(nq 2). These methods are called the basis selection methods. Despite algorithmic benefits, most of the basis selection methods require the assumption that the sample is uniformly-distributed on a hyper-cube. These methods may have deteriorating performance when such an assumption is not met. To overcome the obstacle, we develop an efficient algorithm that is adaptive to the unknown probability density function of the predictors. Theoretically, we show the proposed estimator has the same convergence rate as the full-basis estimator when q is roughly at the order of O[n 2d/{(pr+1)(d +2)}] , where p ∈[1, 2] and r ≈ 4 are some constants depend on the type of the spline. Numerical studies on various synthetic datasets demonstrate the superior performance of the proposed estimator in comparison with mainstream competitors.

A penalized spline fitting method to optimize geometric parameters of arterial centerlines extracted from medical images

In order to grasp the spatial and temporal evolution of vascular geometry, three-dimensional (3D) arterial bending structure and geometrical changes of arteries and stent grafts (SG) must be quantified using geometrical parameters such as curvature and torsion along the vasculature centerlines extracted from medical images. Here, we develop a robust method for constructing smooth centerlines based on a spline fitting method (SFM) such that the optimized geometric parameters of curvature and torsion can be obtained independently of digitization noise in the images. Conventional SFM consists of the 3rd degree spline basis function and 2nd derivative penalty term. In contrast, the present SFM uses the 5th degree spline basis function and 3rd and 4th derivative penalty terms, the coefficients of which are derived by the Akaike information criterion. The results show that the developed SFM can reduce the errors of curvature and torsion compared to conventional SFM. We then apply the present SFM to the centerline of the SG in an abdominal aortic aneurysm (AAA), and those of bilateral internal carotid arteries (ICA) in 6 cases: 3 cases with aneurysms and 3 cases without any aneurysm. The SG centerlines were obtained from temporal medical images at three scan times. The strong peak of the curvature could be clearly observed in the distal area of the SG, the inversion of the torsion at 0 months in the middle area of SG disappeared over time, and the torsions around the SG bifurcation at the three time periods were inverted. The curvature-torsion graphs along the ICA centerlines superimposing five aneurysmal positions were useful for investigating the relationship between arterial bending structure and aneurysmal positions. Both ICAs had curvature peak values higher than 0.4 within the ICA syphons. The ICA torsion graphs indicated that left and right ICA tended to be a right- and left-handed helix, respectively. In the left ICA syphon, the biggest aneurysm could be observed downstream of the salient torsion inversion. All aneurysms for 3 cases were positioned at the downstream of the inverted torsion.

More efficient approximation of smoothing splines via space-filling basis selection

We consider the problem of approximating smoothing spline estimators in a nonparametric regression model. When applied to a sample of size [Formula: see text], the smoothing spline estimator can be expressed as a linear combination of [Formula: see text] basis functions, requiring [Formula: see text] computational time when the number [Formula: see text] of predictors is two or more. Such a sizeable computational cost hinders the broad applicability of smoothing splines. In practice, the full-sample smoothing spline estimator can be approximated by an estimator based on [Formula: see text] randomly selected basis functions, resulting in a computational cost of [Formula: see text]. It is known that these two estimators converge at the same rate when [Formula: see text] is of order [Formula: see text], where [Formula: see text] depends on the true function and [Formula: see text] depends on the type of spline. Such a [Formula: see text] is called the essential number of basis functions. In this article, we develop a more efficient basis selection method. By selecting basis functions corresponding to approximately equally spaced observations, the proposed method chooses a set of basis functions with great diversity. The asymptotic analysis shows that the proposed smoothing spline estimator can decrease [Formula: see text] to around [Formula: see text] when [Formula: see text]. Applications to synthetic and real-world datasets show that the proposed method leads to a smaller prediction error than other basis selection methods.

PRESEE: an MDL/MML algorithm to time-series stream segmenting

Time-series stream is one of the most common data types in data mining field. It is prevalent in fields such as stock market, ecology, and medical care. Segmentation is a key step to accelerate the processing speed of time-series stream mining. Previous algorithms for segmenting mainly focused on the issue of ameliorating precision instead of paying much attention to the efficiency. Moreover, the performance of these algorithms depends heavily on parameters, which are hard for the users to set. In this paper, we propose PRESEE (parameter-free, real-time, and scalable time-series stream segmenting algorithm), which greatly improves the efficiency of time-series stream segmenting. PRESEE is based on both MDL (minimum description length) and MML (minimum message length) methods, which could segment the data automatically. To evaluate the performance of PRESEE, we conduct several experiments on time-series streams of different types and compare it with the state-of-art algorithm. The empirical results show that PRESEE is very efficient for real-time stream datasets by improving segmenting speed nearly ten times. The novelty of this algorithm is further demonstrated by the application of PRESEE in segmenting real-time stream datasets from ChinaFLUX sensor networks data stream.

Locally Adaptive Bayes Nonparametric Regression via Nested Gaussian Processes

We propose a nested Gaussian process (nGP) as a locally adaptive prior for Bayesian nonparametric regression. Specified through a set of stochastic differential equations (SDEs), the nGP imposes a Gaussian process prior for the function's mth-order derivative. The nesting comes in through including a local instantaneous mean function, which is drawn from another Gaussian process inducing adaptivity to locally-varying smoothness. We discuss the support of the nGP prior in terms of the closure of a reproducing kernel Hilbert space, and consider theoretical properties of the posterior. The posterior mean under the nGP prior is shown to be equivalent to the minimizer of a nested penalized sum-of-squares involving penalties for both the global and local roughness of the function. Using highly-efficient Markov chain Monte Carlo for posterior inference, the proposed method performs well in simulation studies compared to several alternatives, and is scalable to massive data, illustrated through a proteomics application.

Automated spectral smoothing with spatially adaptive penalized least squares

A variety of data smoothing techniques exist to address the issue of noise in spectroscopic data. The vast majority, however, require parameter specification by a knowledgeable user, which is typically accomplished by trial and error. In most situations, optimized parameters represent a compromise between noise reduction and signal preservation. In this work, we demonstrate a nonparametric regression approach to spectral smoothing using a spatially adaptive penalized least squares (SAPLS) approach. An iterative optimization procedure is employed that permits gradual flexibility in the smooth fit when statistically significant trends based on multiscale statistics assuming white Gaussian noise are detected. With an estimate of the noise level in the spectrum the procedure is fully automatic with a specified confidence level for the statistics. Potential application to the heteroscedastic noise case is also demonstrated. Performance was assessed in simulations conducted on several synthetic spectra using traditional error measures as well as comparisons of local extrema in the resulting smoothed signals to those in the true spectra. For the simulated spectra, a best case comparison with the Savitzky-Golay smoothing via an exhaustive parameter search was performed while the SAPLS method was assessed for automated application. The application to several dissimilar experimentally obtained Raman spectra is also presented.

Semiparametric regression during 2003-2007

Semiparametric regression is a fusion between parametric regression and nonparametric regression that integrates low-rank penalized splines, mixed model and hierarchical Bayesian methodology - thus allowing more streamlined handling of longitudinal and spatial correlation. We review progress in the field over the five-year period between 2003 and 2007. We find semiparametric regression to be a vibrant field with substantial involvement and activity, continual enhancement and widespread application.

Soft and hard classification by reproducing kernel Hilbert space methods

Reproducing kernel Hilbert space (RKHS) methods provide a unified context for solving a wide variety of statistical modelling and function estimation problems. We consider two such problems: We are given a training set [yi, ti, i = 1, em leader, n], where yi is the response for the ith subject, and ti is a vector of attributes for this subject. The value of y(i) is a label that indicates which category it came from. For the first problem, we wish to build a model from the training set that assigns to each t in an attribute domain of interest an estimate of the probability pj(t) that a (future) subject with attribute vector t is in category j. The second problem is in some sense less ambitious; it is to build a model that assigns to each t a label, which classifies a future subject with that t into one of the categories or possibly "none of the above." The approach to the first of these two problems discussed here is a special case of what is known as penalized likelihood estimation. The approach to the second problem is known as the support vector machine. We also note some alternate but closely related approaches to the second problem. These approaches are all obtained as solutions to optimization problems in RKHS. Many other problems, in particular the solution of ill-posed inverse problems, can be obtained as solutions to optimization problems in RKHS and are mentioned in passing. We caution the reader that although a large literature exists in all of these topics, in this inaugural article we are selectively highlighting work of the author, former students, and other collaborators.

Approximation bounds for some sparse kernel regression algorithms

Gaussian processes have been widely applied to regression problems with good performance. However, they can be computationally expensive. In order to reduce the computational cost, there have been recent studies on using sparse approximations in gaussian processes. In this article, we investigate properties of certain sparse regression algorithms that approximately solve a gaussian process. We obtain approximation bounds and compare our results with related methods.