The doctoral dissertations of the former Helsinki University of Technology (TKK) and Aalto University Schools of Technology (CHEM, ELEC, ENG, SCI) published in electronic format are available in the electronic publications archive of Aalto University - Aaltodoc.
Aalto

Modeling of Mutual Dependencies

Arto Klami

Dissertation for the degree of Doctor of Science in Technology to be presented with due permission of the Faculty of Information and Natural Sciences for public examination and debate in Auditorium T2 at Helsinki University of Technology (Espoo, Finland) on the 5th of September, 2008, at 12 noon.

Overview in PDF format (ISBN 978-951-22-9520-3)   [839 KB]
Dissertation is also available in print (ISBN 978-951-22-9519-7)

Abstract

Data analysis means applying computational models to analyzing large collections of data, such as video signals, text collections, or measurements of gene activities in human cells. Unsupervised or exploratory data analysis refers to a subtask of data analysis, in which the goal is to find novel knowledge based on only the data. A central challenge in unsupervised data analysis is separating relevant and irrelevant information from each other. In this thesis, novel solutions to focusing on more relevant findings are presented.

Measurement noise is one source of irrelevant information. If we have several measurements of the same objects, the noise can be suppressed by averaging over the measurements. Simple averaging is, however, only possible when the measurements share a common representation. In this thesis, we show how irrelevant information can be suppressed or ignored also in cases where the measurements come from different kinds of sensors or sources, such as video and audio recordings of the same scene.

For combining the measurements, we use mutual dependencies between them. Measures of dependency, such as mutual information, characterize commonalities between two sets of measurements. Two measurements can hence be combined to reduce irrelevant variation by finding new representations for the objects so that the representations are maximally dependent. The combination is optimal, given the assumption that what is in common between the measurements is more relevant than information specific to any one of the sources.

Several practical models for the task are introduced. In particular, novel Bayesian generative models, including a Bayesian version of the classical method of canonical correlation analysis, are given. Bayesian modeling is especially justified approach to learning from small data sets. Hence, generative models can be used to extract dependencies in a more reliable manner in, for example, medical applications, where obtaining a large number of samples is difficult. Also, novel non-Bayesian models are presented: Dependent component analysis finds linear projections which capture more general dependencies than earlier methods.

Mutual dependencies can also be used for supervising traditional unsupervised learning methods. The learning metrics principle describes how a new distance metric focusing on relevant information can be derived based on the dependency between the measurements and a supervising signal. In this thesis, the approximations and optimization methods required for using the learning metrics principle are improved.

This thesis consists of an overview and of the following 7 publications:

  1. Arto Klami and Samuel Kaski. 2005. Non-parametric dependent components. In: Proceedings of the 30th IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP 2005). Philadelphia, PA, USA. 18-23 March 2005. Piscataway, NJ, IEEE, pages V-209 - V-212. © 2005 IEEE. By permission.
  2. Abhishek Tripathi, Arto Klami, and Samuel Kaski. 2008. Simple integrative preprocessing preserves what is shared in data sources. BMC Bioinformatics, volume 9, 111. © 2008 by authors.
  3. Arto Klami and Samuel Kaski. 2006. Generative models that discover dependencies between data sets. In: S. McLoone, T. Adali, J. Larsen, and M. Van Hulle (editors). Machine Learning for Signal Processing XVI. Piscataway, NJ, IEEE, pages 123-128. © 2006 IEEE. By permission.
  4. Arto Klami and Samuel Kaski. 2008. Probabilistic approach to detecting dependencies between data sets. Neurocomputing, to appear. © 2008 by authors and © 2008 Elsevier Science. By permission.
  5. Arto Klami and Samuel Kaski. 2007. Local dependent components. In: Zoubin Ghahramani (editor). Proceedings of the 24th International Conference on Machine Learning (ICML 2007). Corvallis, OR, USA. 20-24 June 2007. Madison, WI, Omnipress, pages 425-433. © 2007 by authors.
  6. Jaakko Peltonen, Arto Klami, and Samuel Kaski. 2004. Improved learning of Riemannian metrics for exploratory analysis. Neural Networks, volume 17, numbers 8-9, pages 1087-1100. © 2004 Elsevier Science. By permission.
  7. Samuel Kaski, Janne Sinkkonen, and Arto Klami. 2005. Discriminative clustering. Neurocomputing, volume 69, numbers 1-3, pages 18-41. © 2005 Elsevier Science. By permission.

Errata of publication 6

Keywords: canonical correlation analysis, clustering, data fusion, exploratory data analysis, probabilistic modeling, learning metrics, mutual dependency, mutual information

This publication is copyrighted. You may download, display and print it for Your own personal use. Commercial use is prohibited.

© 2008 Helsinki University of Technology

Last update 2011-05-26