A Geometric Approach to The Unification of Symbolic Structures and Neural Networks

Given a task, symbolists develop rules among symbols, while connectionists construct networks to capture association relations among entities. Being robust to noisy input data, connectionist approaches, in particular Deep-Learning, have been widely adopted in a variety of real applications. Connectionists have been struggling over decades to achieve reasoning quality at the level of symbolic approaches. Alternatively, we propose to create, in continuous space, a ball configuration that precisely spatializes the semantics of symbolic structures, meanwhile well-preserving vector embeddings from connectionist network. Given symbolic tree structures and vector embeddings of tree nodes provided by symbolists and by connectionists, respectively, we have developed geometric transformation algorithms that promote each vector into a ball, namely N-Ball, in higher dimension space under the conditions as follows: (1) each vector is part of the central point of its N-Ball; (2) symbolic tree structures are precisely spatialized, in terms of inclusion relations among N-Balls. With this geometric construction, we unify symbolic structures and neural networks. Such unification is produced by a geometric process that constructs configurations above connectionist model to spatialize symbolic structures. So comes the name Geometric Connectionist Machine (GCM). Our experiments show that N-Ball configurations can be created only in a space with higher dimension than the vector space provided by connectionists and only by abandoning the back-propagation methods. The existence of N-Ball configurations favors limitivism that connectionist network can approximate good symbolic description within certain limit, also favors hybridism that a patchwork, here GCM, can be created to bridge the gap between symbolic and connectionist approaches. A series of experiments using large-scale datasets have shown that a ball configuration significantly outperforms its connectionist component in a number of benchmark tasks. We revisit symbolic-subsymbolic debates in the literature, and conclude that Geometric Connectionist Machines resolve the antagonism between Connectionism and Symbolicism, and in a way create a continuum between connectionist and symbolic models.