Bayesian Regularisation in Model Selection

Gavin Cawley

University of East Anglia, UK

While the model parameters of a kernel machine are typically given by the solution of a convex optimisation problem, with a single global optimum, the selection of good values for the regularisation and kernel parameters is much less straightforward. Fortunately the leave-one-out cross-validation procedure can be performed, or a least approximated, very efficiently in closed form for a wide variety of kernel learning methods, providing a convenient means for model selection. Leave-one-out cross-validation based estimates of performance, however, generally exhibit a relatively high variance and are therefore prone to over-fitting. In this paper, we investigate the use of Bayesian regularisation at the second level of inference, adding a regularisation term to the model selection criterion, corresponding to a prior over the values of the kernel parameters. In this case the additional regularisation parameters at the second level of inverence are integrated out analytically to obviate the need for a higher level selection process. Results obtained on a suite of thirteen real-world and synthetic benchmark datasets clearly demonstrate the benefit of this approach, giving performance comparable to the Expectation Propagation (EP) based Gaussian Process Classifier (GPC).