Andrew Gordon Wilson and Hannes Nickisch:

Kernel Interpolation for Scalable Structured Gaussian Processes

ICML 2015

This paper was clearly one of my highlights at ICML and falls into the category of large-scale kernel machines, one of the trends at ICML. Wilson and Nickisch combine the advantages of inducing point and structure-exploiting (e.g., Kronecker/Toeplitz) approaches.

The key idea behind structured kernel interpolation is to (a) describe current inducing points approaches as a global kernel interpolation, (b) use local interpolation strategies to approximate the cross-covariance between inducing and “real” training points, which allows them to get rid of the requirement of placing inducing points on a grid. This allows us to say something about how the quality of an inducing point approach depends on the number of inducing (=interpolation) points, interpolation strategy, and the GP kernel.

Let us have a look at how the kernel-interpolation approach is derived: In sparse GPs (based on inducing inputs methods), we often compute expressions like

To approximate the kernel matrix. The cross-covariance still depends on the number of data points, i.e., computing this quantity is expensive.

Wilson and Nickisch propose an interpolation-based approach to approximate . As an illustration, let’s assume that a data point lies “between” inducing inputs . This motivates a possible approximation of the cross-covariance by a convex combination , which no longer requires explicitly. This leads to the central approximation in this paper:

where is a matrix of interpolation weights that are determined based on the distances between the training points and the inducing inputs . Note that this matrix can be very sparse. With this approach we arrive at the alternative approximation of the kernel matrix:

.

This approximation can be combined with structure-exploiting approaches (e.g., Toeplitz, Kronecker).

Even without structure-exploitation, the computational complexity is , an improvement over the typical of sparse GP methods based on inducing points.

The kernel interpolation approach is an interesting approach to sparse GPs. It seems that currently we still depend on some sort of “distance”, which allows us to compute the weight matrix , i.e., this approach might not directly work with the full breadth of kernels that are available. However, it might be possible to learn an inducing input representation that addresses this problem; and potentially the inducing inputs may lie in a significantly lower-dimensional space than the original inputs. Let’s see what follow-ups will come.