PhD Defence | Inductive Biases for Pixel Representation Learning

A machine-learning algorithm with an ability to generalize beyond the data it has seen during training must have an inductive bias. An inductive bias allows a learning algorithm to prioritize one solution over another, and to generalize beyond the training data. Several inductive biases have been successfully used in classical machine learning methods. For example, linear regression assumes that the target has a linear relationship with each of the input features. Inductive biases have also been explored for deep learning, and this is what in part contributed to its success. Besides the well-known explicit inductive biases like L1 \& L2 regularization, there are also many implicit inductive biases, which are introduced by implicit regularization such as optimization algorithms, dropout, attention mechanism, transfer learning. These implicit inductive biases have been showing remarkable ability to help generalization in deep learning. Different from explicit inductive biases, however, it is hard to explicitly identify and derive a formulation of implicit bias. Also, there is no direct way to control the strength of an implicit bias. As a result, it is difficult to apply an implicit bias in practice.

Zenglin’s thesis strives to turn the implicit biases into an explicit form to uncover and exploit them for pixel representation learning. This research has uncovered and exploited three implicit biases for different pixel-level tasks, including the spectral bias for the deep image prior, the salience bias for guided filtering, and the attentional bias for tiny object localization and counting. Furthermore, this thesis also seeks to develop new inductive biases by exploiting prior knowledge and has developed three inductive biases for best-in-class object counting by discovering new knowledge. Overall, Zenglin’s research has shown the importance of uncovering and exploiting inductive biases in improving generalization for pixel representation learning.