PhD Thesis in Machine Learning for Personalized Medicine Bringing Models to the Domain: Deploying Gaussian Processes in the Biological Sciences Max Zwießele October 12, 2017 Supervisor Neil D. Lawrence Magnus Rattray University of Sheffield University of Manchester Name, first name: Zwießele, Max Registration No: 120261372 Department of Computer Science Bringing Models to the Domain: Deploying Gaussian Processes in the Biological Sciences PhD Thesis University of Sheffield Period: 01.04.2013-31.3.2017 ii Thesis Deposit Agreement 1. I, the author, confirm that the Thesis is my own work, and that where materi- als owned by a third party have been used, copyright clearance has been ob- tained. I am aware of the University’s Guidance on the Use of Unfair Means (www.sheffield.ac.uk/ssid/exams/plagiarism). 2. I confirm that all copies of the Thesis submitted to the University, whether in print or electronic format, are identical in content and correspond with the version of the Thesis upon which the examiners based their recommendation for the award of the degree (unless edited as indicated above). 3. I agree to the named Thesis being made available in accordance with the con- ditions specified above. 4. I give permission to the University of Sheffield to reproduce the print Thesis (where applicable) in digital format, in whole or part, in order to supply sin- gle copies for the purpose of research or private study for a non-commercial purpose. I agree that a copy of the eThesis may be supplied to the British Library for inclusion on EThOS and WREO, if the thesis is not subject to an embargo, or if the embargo has been lifted or expired. 5. I agree that the University of Sheffield’s eThesis repository (currently WREO) will make my eThesis (where applicable) available over the internet via an entirely non-exclusive agreement and that, without changing content, WREO and/or the British Library may convert my eThesis to any medium or format for the purpose of future preservation and accessibility. 6. I agree that the metadata relating to the eThesis (where applicable) will nor- mally appear on both the University’s eThesis server (WREO) and the British Library’s EThOS service, even if the eThesis is subject to an embargo. Signature Date Max Zwiessele iii University stamp iv Abstract Recent developments in single cell sequencing allow us to elucidate processes of individual cells in unprecedented detail. This detail provides new insights into the progress of cells during cell type differentiation. Cell type heterogeneity shows the complexity of cells working together to produce organ function on a macro level. The understanding of single cell transcriptomics promises to lead to the ultimate goal of understanding the function of individual cells and their contribution to higher level function in their environment. Characterizing the transcriptome of single cells requires us to understand and be able to model the latent processes of cell functions that explain biological vari- ance and richness of gene expression measurements. In this thesis, we describe ways of jointly modelling biological function and unwanted technical and biolog- ical confounding variation using Gaussian process latent variable models. In ad- dition to mathematical modelling of latent processes, we provide insights into the understanding of research code and the significance of computer science in devel- opment of techniques for single cell experiments. We will describe the process of understanding complex machine learning al- gorithms and translating them into usable software. We then proceed to applying these algorithms. We show how proper research software design underlying the implementation can lead to a large user base in other areas of expertise, such as single cell gene expression. To show the worth of properly designed software un- derlying a research project, we show other software packages built upon the soft- ware developed during this thesis and how they can be applied to single cell gene expression experiments. v Understanding the underlying function of cells seems within reach through these new techniques that allow us to unravel the transcriptome of single cells. We describe probabilistic techniques of identifying the latent functions of cells, while focusing on the software and ease-of-use aspects of supplying proper research code to be applied by other researchers. vi Acknowledgements First of all, I thank Neil Lawrence for helping me through the process of writing a PhD thesis and providing guidance along the way. Special thanks go to Karsten Borgwardt to sending me on this path, Bertram Müller Myhsok and Volker Tresp for letting me second in their respective research labs. I also thank the members of my work group in Sheffield - Michael Croucher, Zhen- wen Dai, Andreas Damianou, Nicolo Fusi, Javier Gonzalez, James Hensman, Alan Saul and Michael Smith - for useful, inspiring discussion and proper proofreading. Additionally I thank Sarah Teichmann and AlekMsandraaKolocdzihejczyiknfor sueppo rt in biological questions and interpretation of results, as well as providing biological data for analysis. Learning Finally, I thank my parents Sibylle and Frieder for their patience and assistance in helping me finishing this thesis. for I am grateful for financial support from the European Union 7th Framework Pro- gramme through the Marie Curie Initial Training Network “Machine Learning for Personalized Personalized Medicine” MLPM2012, Grant No. 316861. Medicine 0100111010 1101001110 1011010011 1010110100 1110100100 vii viii Contents Abstract v Acknowledgements vii Contents ix List of Figures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiii List of Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xv Nomenclature xvi 1 Introduction 1 1.1 Contribution and Roadmap . . . . . . . . . . . . . . . . . . . . . . . . . . 3 1.2 Biological Background - From DNA to Protein . . . . . . . . . . . . . . . 4 1.2.1 Discovery and Structure of DNA . . . . . . . . . . . . . . . . . . . 4 1.2.2 Functional View on DNA – Genes, Expression and Proteins . . 6 1.3 Machine Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 1.4 Research Code . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 1.4.1 Tutorials . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 1.4.2 Codebase & Knowledge of Algorithm . . . . . . . . . . . . . . . 11 1.4.3 Making Algorithms Accessible . . . . . . . . . . . . . . . . . . . . 12 1.4.4 Github . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 1.4.5 Automating Code Correctness . . . . . . . . . . . . . . . . . . . . 13 1.4.6 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 ix 2 Methods 15 2.1 Gaussian Process Regression . . . . . . . . . . . . . . . . . . . . . . . . . . 16 2.1.1 Gradients . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 2.1.2 Gaussian Process Prior (Covariance Function) . . . . . . . . . . 20 2.1.3 Prediction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24 2.1.4 Example and Sample . . . . . . . . . . . . . . . . . . . . . . . . . . 24 2.1.5 ARD: Automatic Relevance Determination . . . . . . . . . . . . 25 2.2 Principal Component Analysis . . . . . . . . . . . . . . . . . . . . . . . . 27 2.3 Gaussian Process Latent Variable Model (GPLVM) . . . . . . . . . . . . 30 2.3.1 Inferring Subspace Dimensionality . . . . . . . . . . . . . . . . . 32 2.4 Sparse Gaussian Process Regression . . . . . . . . . . . . . . . . . . . . . 33 2.4.1 Optimization and Complexity . . . . . . . . . . . . . . . . . . . . 35 2.4.2 Implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35 2.4.3 Prediction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35 2.4.4 Intuition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36 2.5 Variational Bayesian GPLVM . . . . . . . . . . . . . . . . . . . . . . . . . . 37 2.5.1 On ARD Parameterization in Bayesian GPLVM . . . . . . . . . 38 2.5.2 Implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39 2.5.3 Factorization and Parallelization . . . . . . . . . . . . . . . . . . . 39 2.5.4 Large Scale Bayesian GPLVM . . . . . . . . . . . . . . . . . . . . . 41 2.6 MRD: Manifold Relevance Determination . . . . . . . . . . . . . . . . . 42 2.6.1 Intuition and Simulation . . . . . . . . . . . . . . . . . . . . . . . . 43 3 Case Study GPy 49 3.1 Splitting the Algorithm into Parts . . . . . . . . . . . . . . . . . . . . . . . 50 3.2 Likelihood & Prior . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51 3.3 Inference . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51 3.4 Numerical Stability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52 3.5 Posterior Prediction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54 3.6 Gradients . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55 3.7 Optimization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56 x