Singel 411, 1012 WN Amsterdam, Netherlands
PhD Defence | A Coding Perspective on Deep Latent Variable Models
Karen Ullrich’s thesis “A Coding Perspective on Deep Latent Variable Models” discusses how statistical inference in Deep Latent Variable Models (DLVMs) relates to coding. Ullrich completed her research under the supervision of Max Welling and Patrick Forré (both from the UvA).
In particular, Ullrich examines the minimum deception length (MDL) principle as a guide for statistical inference. In this context, she explores its relation to Bayesian inference. It can be observed that despite both leading to similar algorithms, the MDL principle allows us to make no assumption about the data generating process. We merely restrict ourselves to finding regularity in the observed data, where regularity is connected to the ability to compress. She thus finds that learning DLVMs is equivalent to minimizing the cost for communicating (compressing) a set of observations. One common approach to communication is to send a hypothesis (or model), and subsequently the data misfit under the aforementioned model. This is known as the two-part code. In her thesis, Ullrich will mainly focus on the so-called Bayesian code – a theoretically more effective code than the two-part code.
Somewhat counter-intuitively, the Bayesian inference method will allow us to compute the code length without knowing the code nor the coding scheme that achieved this code length. The purpose of the thesis is to close this gap by developing respective coding schemes. Ullrich will, inspired and guided by the MDL principle, look for the codes that achieve the code length predicted by MDL. A special focus lies on differentiable functions, and more precisely, deep neural networks, learned by way of large quantities of high dimensional data. She will investigate model compression as well as source compression through the lens of the MDL principle.