PhD Defence | Machine Learning on Multimodal Knowledge Graphs
The knowledge graph is a data model in which knowledge, information, and data are all encoded in graph form using the same basic building blocks. This knowledge can be entirely made up of objects, expressing all information through their connectivity, but knowledge graphs are also capable of seamlessly integrating other forms of information, including images, natural language, and spatial information, making the knowledge graph a suitable choice to model heterogeneous knowledge with: information of different types and from different domains. With a wealth of heterogeneous knowledge already available in knowledge graph format, and with the expectation that this amount is only to grow in the future, the knowledge graph data model becomes ever more interesting for machine learning scientists and practitioners to learn on.
In his thesis, Xander Wilcke identifies many of the opportunities and challenges that arise with machine learning on heterogeneous knowledge, encoded as knowledge graph, and investigates 1) how machine learning models can be build that naturally incorporate this heterogeneity and to what extent this affects their performance, and 2) how data scientists can use such models to discover interesting patterns in knowledge graphs that may help experts perform various downstream tasks. These lines are addressed in six chapters and along three dimensions: to what extent 1) contextual and 2) multimodal information are included in the learning process, and 3) the level of involvement of experts in this process. Special attention is given to spatial information, such as coordinates and geometries, which is an integral component of many real-world datasets.