Research Article: Functional form estimation using oblique projection matrices for LS-SVM regression models

Date Published: June 7, 2019

Publisher: Public Library of Science

Author(s): Alexander Caicedo, Carolina Varon, Sabine Van Huffel, Johan A. K. Suykens, Antonio Agudo.


Kernel regression models have been used as non-parametric methods for fitting experimental data. However, due to their non-parametric nature, they belong to the so-called “black box” models, indicating that the relation between the input variables and the output, depending on the kernel selection, is unknown. In this paper we propose a new methodology to retrieve the relation between each input regressor variable and the output in a least squares support vector machine (LS-SVM) regression model. The method is based on oblique subspace projectors (ObSP), which allows to decouple the influence of input regressors on the output by including the undesired variables in the null space of the projection matrix. Such functional relations are represented by the nonlinear transformation of the input regressors, and their subspaces are estimated using appropriate kernel evaluations. We exploit the properties of ObSP in order to decompose the output of the obtained regression model as a sum of the partial nonlinear contributions and interaction effects of the input variables, we called this methodology Nonlinear ObSP (NObSP). We compare the performance of the proposed algorithm with the component selection and smooth operator (COSSO) for smoothing spline ANOVA models. We use as benchmark 2 toy examples and a real life regression model using the concrete strength dataset from the UCI machine learning repository. We showed that NObSP is able to outperform COSSO, producing stable estimations of the functional relations between the input regressors and the output, without the use of prior-knowledge. This methodology can be used in order to understand the functional relations between the inputs and the output in a regression model, retrieving the physical interpretation of the regression models.

Partial Text

Non-parametric regression is an important field of data analysis. These non-parametric models use some observations of the input data and the desired target to estimate a function and make predictions [1, 2]. However, generally, these models focus on the prediction of the target variable of interest and not on the model interpretability. In this manuscript, we will refer to interpretability as the property of a model to express the output in additive terms of the partial nonlinear contributions of the input variables and their interaction effects. In several applications interpretability plays an important role in the construction of prediction models. In such cases, the main goal is not the prediction of the response of a system but to determine the underlying mechanisms and the relationship between the inputs and the output. For instance, in medical applications, this information can be used in order to identify treatment targets, support diagnosis, and facilitate the introduction of these models in clinical practice. As an example, Van Belle et al. proposed the use of a color code to enhance the interpretability of classification models for the clinicians [3].

LS-SVM is a kernel based methodology that can be used to solve nonlinear classification and regression problems [18]. Due to its flexibility to manage different kind of problems and produce an adequate mathematical model, LS-SVM has been used successfully in different application fields such as: the prediction of electricity energy consumption [19], estimation of water pollution [20], forecasting of carbon price [21], and the prediction of meteorological time series [22], among others. However, its applications have been hampered due to its black-box model nature.

In this section we present the propossed decomposition algorithm, which is a nonlinear extention to oblique subspace projections (NObSP). NObSP is not a regression method but an algorithm that allows to decompose the output of a regression model, using LS-SVM regression, into additive components that represent the partial nonlinear contributions of the input regressors on the output, and their interaction effects. NObSP uses oblique subspace projections to extract the partial contribution of an input regressor, while nullifying the contributions from other regressors. We will first introduce the concept of oblique subspace projections, then we will proposed their extention for nonlinear regression models.

In this section we present results from the proposed algorithm using 2 simulation examples, as well as a real life example using data from the Concrete Compressive Strength Data Set in the UCI machine learning repository [25]. In the first toy example we compare NObSP with the results given by COSSO, using the same example proposed in [11], where only main effects are included. Additionally, we evaluate the performance of NObSP to select relevant components, and we test the effect of the kernel selection in the decomposition. In the second example, we create an artificial dataset that includes interaction effects, we test for the robustness of the projections and the regression model by means of bootstrapping, we present the results in terms of their mean solution and 95% confidence intervals, in this example we also evaluated the performance of the model to unseen test data. In the third example we demonstrate the potential use of NObSP in a real life example.

We have proposed an algorithm to decompose the output of an LS-SVM regression model into the sum of the partial nonlinear contributions, as well as interaction effects, of the input regressors. We have demonstrated that the functional form of the relation between the inputs and the output can be retrieved using oblique subspace projections. We have also shown that through appropriate kernel evaluations it is possible to find a proper basis for the subspace representing the nonlinear transformation of the input regression variables.

In this manuscript we proposed a decomposition scheme using oblique subspace projections, called NObSP, which is able to retrieve relevant information about the input variables and their relationship with the output in an LS-SVM regression model, facilitating its interpretation. The performance of the proposed model was demonstrated using 2 different toy examples as well as a practical example from a public database. This methodology has a huge potential in many fields, including biomedical applications. For instance, it can be used for the study of the interactions between different physiological signals, providing extra information for the understanding of some underlying regulatory mechanisms, and supporting clinical diagnosis and treatment.