The doctoral dissertations of the former Helsinki University of Technology (TKK) and Aalto University Schools of Technology (CHEM, ELEC, ENG, SCI) published in electronic format are available in the electronic publications archive of Aalto University - Aaltodoc.
Aalto

Speeding up the Inference in Gaussian Process Models

Jarno Vanhatalo

Doctoral dissertation for the degree of Doctor of Science in Technology to be presented with due permission of the Faculty of Information and Natural Sciences for public examination and debate in Auditorium E at the Aalto University School of Science and Technology (Espoo, Finland) on the 19th of October 2010 at 12 noon.

Overview in PDF format (ISBN 978-952-60-3381-5)   [580 KB]
Errata (in PDF format)
Dissertation is also available in print (ISBN 978-952-60-3380-8)

Abstract

In this dissertation Gaussian processes are used to define prior distributions over latent functions in hierarchical Bayesian models. Gaussian process is a non-parametric model with which one does not need to fix the functional form of the latent function, but its properties can be defined implicitly. These implicit statements are encoded in the mean and covariance function, which determine, for example, the smoothness and variability of the function. This non-parametric nature of the Gaussian process gives rise to a flexible and diverse class of probabilistic models.

There are two main challenges with using Gaussian processes. Their main complication is the computational time which increases rapidly as a function of a number of data points. Other challenge is the analytically intractable inference, which exacerbates the slow computational time. This dissertation considers methods to alleviate these problems.

The inference problem is attacked with approximative methods. The Laplace approximation and expectation propagation algorithm are utilized to give Gaussian approximation to the conditional posterior distribution of the latent function given the hyperparameters. The integration over hyperparameters is performed using a Monte Carlo, a grid based, or a central composite design integration. Markov chain Monte Carlo methods over all unknown parameters are used as a golden standard to which the other methods are compared. The rapidly increasing computational time is cured with sparse approximations to Gaussian process and compactly supported covariance functions. These are both analyzed in detail and tested in experiments. Practical details on their implementation with the approximative inference techniques are discussed.

The techniques for speeding up the inference are tested in three modeling problems. The problems considered are disease mapping, regression and classification. The disease mapping and regression problems are tackled with standard and robust observation models. The results show that the techniques presented speed up the inference considerably without compromising the accuracy severely.

This thesis consists of an overview and of the following 6 publications:

  1. Jarno Vanhatalo and Aki Vehtari. 2007. Sparse log Gaussian processes via MCMC for spatial epidemiology. In: Neil Lawrence, Anton Schwaighofer, and Joaquin Quiñonero Candela (editors). Gaussian Processes in Practice. JMLR: Workshop and Conference Proceedings, volume 1, pages 73-89. © 2007 by authors.
  2. Jarno Vanhatalo and Aki Vehtari. 2008. Modelling local and global phenomena with sparse Gaussian processes. In: David A. McAllester and Petri Myllymäki (editors). Proceedings of the 24th Conference on Uncertainty in Artificial Intelligence (UAI 2008). Helsinki, Finland. 9-12 July 2008. Corvallis, Oregon, USA. AUAI Press. Pages 571-578. ISBN 0-9749039-4-9. © 2008 by authors.
  3. Jarno Vanhatalo, Pasi Jylänki, and Aki Vehtari. 2009. Gaussian process regression with Student-t likelihood. In: Yoshua Bengio, Dale Schuurmans, John Lafferty, Chris Williams, and Aron Culotta (editors). Advances in Neural Information Processing Systems. Proceedings of the Twenty-Third Annual Conference on Neural Information Processing Systems (NIPS 2009). Vancouver, BC, Canada. 7-10 December 2009. Red Hook, NY, USA. Curran Associates. Volume 22, pages 1910-1918. ISBN 978-1-615679-11-9. © 2009 by authors.
  4. Jarno Vanhatalo, Ville Pietiläinen, and Aki Vehtari. 2010. Approximate inference for disease mapping with sparse Gaussian processes. Statistics in Medicine, volume 29, number 15, pages 1580-1607. © 2010 John Wiley & Sons. By permission.
  5. Jarno Vanhatalo and Aki Vehtari. 2010. Speeding up the binary Gaussian process classification. In: Peter Grünwald and Peter Spirtes (editors). Proceedings of the 26th Conference on Uncertainty in Artificial Intelligence (UAI 2010). Catalina Island, California, USA. 8-11 July 2010. Corvallis, Oregon, USA. AUAI Press. Pages 623-631. © 2010 by authors.
  6. Jarno Vanhatalo, Pia Mäkelä, and Aki Vehtari. 2010. Regional differences in alcohol mortality in Finland in the early 2000s. Espoo, Finland: Aalto University School of Science and Technology. 12 pages. Helsinki University of Technology, Department of Biomedical Engineering and Computational Science Publications, Report A20. ISBN 978-952-60-3335-8. ISSN 1797-3996. Translation of the original Finnish article: Jarno Vanhatalo, Pia Mäkelä, and Aki Vehtari. 2010. Alkoholikuolleisuuden alueelliset erot Suomessa 2000-luvun alussa. Yhteiskuntapolitiikka, volume 75, number 3, pages 265-273. © 2010 by authors.

Errata of publication 1

Keywords: sparse Gaussian process, approximate inference, compactly supported covariance function

This publication is copyrighted. You may download, display and print it for Your own personal use. Commercial use is prohibited.

© 2010 Aalto University School of Science and Technology

Last update 2014-03-19