Informed Pre-Training of Neural Networks using Prototypes from Prior Knowledge

We present a novel approach for hybrid AI and propose informed pre-training on prototypes from prior knowledge. Generally, when training data is scarce, the incorporation of additional knowledge can assist the learning process of neural networks. An approach that recently gained a lot of interest is informed machine learning, which integrates prior knowledge that is explicitly given by formal representations, such as graphs or equations. However, the integration often is application-specific and can be time-consuming. Another more straightforward approach is pre-training on other large data sets, which allows to reuse knowledge that is implicitly stored in trained models. This raises the question, if it is also possible to pre-train a neural network on a small set of knowledge representations. In this paper, we investigate this idea and propose informed pre-training on knowledge prototypes. Such prototypes are often available and represent characteristic semantics of the domain. We show that it (i) improves generalization capabilities, (ii) increases out-of-distribution robustness, and (iii) speeds up learning. Moreover, we analyze which parts of a neural network model are affected most by our informed pre-training approach. We discover that (iv) improvements come from deeper layers that typically represent high-level features, which confirms the transfer of semantic knowledge. This is a before unobserved effect and shows that informed transfer learning has additional and complementary strengths to existing approaches.