Bayesian Regularisation
in Model Selection
Gavin Cawley
University of East Anglia, UK
While the model parameters of a kernel machine are typically given by the
solution of a convex optimisation problem, with a single global optimum,
the selection of good values for the regularisation and kernel parameters
is much less straightforward. Fortunately the leave-one-out cross-validation
procedure can be performed, or a least approximated, very efficiently in
closed form for a wide variety of kernel learning methods, providing a convenient
means for model selection. Leave-one-out cross-validation based estimates
of performance, however, generally exhibit a relatively high variance and
are therefore prone to over-fitting. In this paper, we investigate the use
of Bayesian regularisation at the second level of inference, adding a regularisation
term to the model selection criterion, corresponding to a prior over the
values of the kernel parameters. In this case the additional regularisation
parameters at the second level of inverence are integrated out analytically
to obviate the need for a higher level selection process. Results obtained
on a suite of thirteen real-world and synthetic benchmark datasets clearly
demonstrate the benefit of this approach, giving performance comparable to
the Expectation Propagation (EP) based Gaussian Process Classifier (GPC).