We present a novel approach for hybrid AI and propose informed pre-training on prototypes from prior knowledge. Generally, when training data is scarce, the incorporation of additional knowledge can assist the learning process of neural networks. An approach that recently gained a lot of interest is informed machine learning, which integrates prior knowledge that is explicitly given by formal representations, such as graphs or equations. However, the integration often is application-specific and can be time-consuming. Another more straightforward approach is pre-training on other large data sets, which allows to reuse knowledge that is implicitly stored in trained models. This raises the question, if it is also possible to pre-train a neural network on a small set of knowledge representations. In this paper, we investigate this idea and propose informed pre-training on knowledge prototypes. Such prototypes are often available and represent characteristic semantics of the domain. We show that it (i) improves generalization capabilities, (ii) increases out-of-distribution robustness, and (iii) speeds up learning. Moreover, we analyze which parts of a neural network model are affected most by our informed pre-training approach. We discover that (iv) improvements come from deeper layers that typically represent high-level features, which confirms the transfer of semantic knowledge. This is a before unobserved effect and shows that informed transfer learning has additional and complementary strengths to existing approaches.<\/p>\n","protected":false},"excerpt":{"rendered":"
We present a novel approach for hybrid AI and propose informed pre-training on prototypes from prior knowledge. Generally, when training data is scarce, the incorporation of additional knowledge can assist the learning process of neural networks. An approach that recently gained a lot of interest is informed machine learning, which integrates prior knowledge that is explicitly given by formal representations, such as graphs or equations. However, the integration often is […]<\/p>\n","protected":false},"author":14,"featured_media":0,"template":"","meta":{"_acf_changed":false,"footnotes":""},"publication-type":[30],"class_list":["post-32903","publication","type-publication","status-publish","hentry","publication-type-article"],"acf":[],"publishpress_future_workflow_manual_trigger":{"enabledWorkflows":[]},"_links":{"self":[{"href":"https:\/\/lamarr-institute.org\/de\/wp-json\/wp\/v2\/publication\/32903","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/lamarr-institute.org\/de\/wp-json\/wp\/v2\/publication"}],"about":[{"href":"https:\/\/lamarr-institute.org\/de\/wp-json\/wp\/v2\/types\/publication"}],"author":[{"embeddable":true,"href":"https:\/\/lamarr-institute.org\/de\/wp-json\/wp\/v2\/users\/14"}],"version-history":[{"count":0,"href":"https:\/\/lamarr-institute.org\/de\/wp-json\/wp\/v2\/publication\/32903\/revisions"}],"wp:attachment":[{"href":"https:\/\/lamarr-institute.org\/de\/wp-json\/wp\/v2\/media?parent=32903"}],"wp:term":[{"taxonomy":"publication-type","embeddable":true,"href":"https:\/\/lamarr-institute.org\/de\/wp-json\/wp\/v2\/publication-type?post=32903"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}