Informed Pre-Training of Neural Networks using Prototypes from Prior Knowledge

We present a novel approach for hybrid AI and propose informed pre-training on prototypes from prior knowledge. Generally, when training data is scarce, the incorporation of additional knowledge can assist the learning process of neural networks. An approach that recently gained a lot of interest is informed machine learning, which integrates prior knowledge that is explicitly given by formal representations, such as graphs or equations. However, the integration often is application-specific and can be time-consuming. Another more straightforward approach is pre-training on other large data sets, which allows to reuse knowledge that is implicitly stored in trained models. This raises the question, if it is also possible to pre-train a neural network on a small set of knowledge representations. In this paper, we investigate this idea and propose informed pre-training on knowledge prototypes. Such prototypes are often available and represent characteristic semantics of the domain. We show that it (i) improves generalization capabilities, (ii) increases out-of-distribution robustness, and (iii) speeds up learning. Moreover, we analyze which parts of a neural network model are affected most by our informed pre-training approach. We discover that (iv) improvements come from deeper layers that typically represent high-level features, which confirms the transfer of semantic knowledge. This is a before unobserved effect and shows that informed transfer learning has additional and complementary strengths to existing approaches.

  • Published in:
    International Joint Conference on Neural Networks
  • Type:
    Inproceedings
  • Authors:
    von Rueden, Laura; Houben, Sebastian; Cvejoski, Kostadin; Garcke, Jochen; Bauckhage, Christian; Piatkowski, Nico
  • Year:
    2023

Citation information

von Rueden, Laura; Houben, Sebastian; Cvejoski, Kostadin; Garcke, Jochen; Bauckhage, Christian; Piatkowski, Nico: Informed Pre-Training of Neural Networks using Prototypes from Prior Knowledge, International Joint Conference on Neural Networks, 2023, https://arxiv.org/abs/2205.11433, Rueden.etal.2023a,

Associated Lamarr Researchers

lamarr institute person von Rueden Laura Platzhalter - Lamarr Institute for Machine Learning (ML) and Artificial Intelligence (AI)

Laura von Rueden

Autorin to the profile
lamarr institute person Houben Sebastian - Lamarr Institute for Machine Learning (ML) and Artificial Intelligence (AI)

Dr. Sebastian Houben

Author to the profile
lamarr institute person Bauckhage Christian - Lamarr Institute for Machine Learning (ML) and Artificial Intelligence (AI)

Prof. Dr. Christian Bauckhage

Director to the profile
lamarr institute person Piatkowski Nico - Lamarr Institute for Machine Learning (ML) and Artificial Intelligence (AI)

Dr. Nico Piatkowski

Autor to the profile