![[Gaussian Process Summer School Logo]](/gpss26/assets/images/logo-gpss.svg){kind=logo}
Day 3 Advances in Gaussian process models |
Room: Engineering Building 2B025
8:30-9:00 Arrivals
9:00-9:10 Welcome
9:10-10:10 Learning the Spectrum, Not the Kernel: Expressive Spectral Densities for Multi-Output GPs and Attention
10:10-10:40 Coffee Break
10:40-11:40 Forgetting to Improve: Principled Data Removal in Active Learning
11:40-12:40 Gaussian Process Variational Autoencoders
12:40-14:00 Lunch
14:00-15:00 Scalable Gaussian Processes on High-Dimensional Incomplete Grids
15:00-15:30 Tea Break
15:30-16:30 Transformed Latent Variable MultiOutput Gaussian Processes
Abstracts
Learning the Spectrum, Not the Kernel: Expressive Spectral Densities for Multi-Output GPs and Attention
The first lecture treated a kernel's spectral density as a learnable object, optimized from data while preserving scalable random-feature approximations. This lecture carries that principle into two settings where the kernel must do more work, asking in each how flexible the spectral density can be made before duality, scalability, or parameterization breaks down. We open briefly with multi-output Gaussian processes, where the kernel becomes matrix-valued and must capture both per-output structure and cross-output dependence. Revisiting the spectral-kernel duality here shows how the conditions for a valid kernel push existing designs into over-parameterized yet inflexible forms, and how relaxing them—with a low-rank construction that gives each output a latent spectral embedding—restores expressiveness at linear cost. The remainder focuses on attention. Viewing self-attention as a kernel smoother over tokens, with softmax attention as one special case, we cast nonstationary attention as bivariate spectral density learning. This reuses random Fourier features for linear-time computation and normalizing flows—introduced in the first lecture—to parameterize an explicit, regularizable density, unlike mixture-based or implicit neural alternatives that are costly or lack a density to regularize. We then show how a sufficiently expressive learned kernel reshapes the architecture itself: it can absorb the query and key projections and fix the value projections to be orthogonal, yielding a markedly more parameter-efficient attention module that retains adaptive, input-dependent similarity. The closing message is that learning the spectrum, rather than the kernel directly, is a portable design principle that travels from GP regression to deep sequence models.
Forgetting to Improve: Principled Data Removal in Active Learning
The uncertainty of a statistical model is most commonly factorized into an aleatoric and an epistemic part. This factorization changes how predictions are interpreted in downstream decision tasks. Importantly, except for the idealistic scenario with no model mismatch, the quantification is a characteristic of the model and not the data generating process. In this paper, we propose Forgetting to Improve a method that reduces this discrepancy by incorporating the task into the modeling framework. Our key insight is to acknowledge that in scenarios of model mismatch, data can have a detrimental effect on the modeling for a specific task. Based on this insight, we propose an influence function for Gaussian process models that allows for principled removal of detrimental data samples. We showcase the flexibility of this approach by demonstrating significant improvements across a range of tasks, including Bayesian optimization, model-based reinforcement learning, and transductive learning.
Gaussian Process Variational Autoencoders
Gaussian processes (GPs) provide a principled way to encode structure, uncertainty, and inductive bias in latent-variable models, while Variational Autoencoders (VAEs) offer scalable amortised inference for complex, high-dimensional observations. In this talk, I will introduce Gaussian Process Variational Autoencoders (GPVAEs), a family of models that combine these two perspectives by placing GP-based structure in the latent space of deep generative models. The talk will begin with the motivation for GPVAEs, emphasising the role of GP priors in structured latent-variable modelling. It will then review representative GPVAE models in terms of their modelling assumptions, inference strategies, and computational trade-offs. The final part will present our work on Nearest Neighbour GPVAE, a scalable GPVAE framework based on neighbour-driven GP approximations.
Scalable Gaussian Processes on High-Dimensional Incomplete Grids
In science, one is frequently faced with the task of modelling high-dimensional functions, e.g., potential energy surfaces (PESs) which play a crucial role in chemistry. Gaussian process regression (GPR) is an attractive tool for modelling such functions, but scalability remains a challenge. It is possible to vastly reduce the cost of doing GPR by exploiting structure in the kernel matrix. Existing grid-based methods exploit Kronecker structure in the kernel, which leads to near-linear complexity with the number of training points (N). Unfortunately, the computational cost scales exponentially with the number of input dimensions (D), which is a severe limitation. The source of the exponential scaling is the reliance on complete Cartesian product grids, so the question arises: Can we keep the benefits of grid structure without committing to complete grids? We have recently introduced CUTS-GPR, a new method for performing numerically exact Gaussian process regression (GPR) in high-dimensional settings. The key component of CUTS-GPR is an extremely fast kernel matrix-vector product, which exhibits near-linear or even linear scaling with N and low-order polynomial scaling D. This is obtained by combining an additive kernel with an incomplete grid and exploiting the resulting structure of the kernel matrix. Using iterative methods, which have become standard in the GP community, the matrix-vector is sufficient to do complete GPR calculations, including hyperparameter optimisation and predictions.
Transformed Latent Variable MultiOutput Gaussian Processes
MultiOutput Gaussian Processes (MOGPs) provide a principled probabilistic framework for modelling correlated outputs but face scalability bottlenecks when applied to datasets with high dimensional output spaces. To maintain tractability, existing methods typically resort to restrictive assumptions, such as employing low rank or sum of separable kernels, which can limit expressiveness. We propose the Transformed Latent Variable MOGP (TLVMOGP), a novel framework that scales MOGPs to a massive number of outputs while preserving the capacity to capture meaningful interoutput dependencies. TLVMOGP constructs a flexible multioutput deep kernel by mapping inputs and output specific latent variables into an embedding space using a Lipschitz regularised neural network. Combined with stochastic variational inference, our model effectively scales to high dimensional output settings. Across diverse benchmarks, including climate modelling with over 10000 outputs and zero inflated spatial transcriptomics data, TLVMOGP outperforms baselines in both predictive accuracy and computational efficiency.