Integrating Neural Networks with Symbolic AI: A Methodology

2020-01-16

The recent commercial AI revolution is largely driven by deep neural networks. Deep neural networks were first invented in the 1960s and began to develop once combined with large-scale datasets and distributed GPU computing.


However, there is more than one algorithm in the field of AI. Symbolic reasoning algorithms, such as artificial intelligence systems, which were also pioneered in the 1960s, may also become a focus of attention.



Weaknesses of Deep Neural Networks


Deep neural networks perform exceptionally well on certain tasks, such as image recognition and machine translation. However, for many more complex applications, traditional deep learning methods cannot match the capabilities of hybrid architecture systems, which also require the use of other AI technologies such as probabilistic reasoning, seed ontology, and self-reprogramming abilities.


Deep neural networks themselves lack strong generalization, meaning the ability to discover new patterns and make inferences beyond the training set. Deep neural networks interpolate and approximate known information, so they cannot truly possess human creativity.


This is why a large amount of training data is needed to teach deep neural networks, and why data augmentation is such an important deep learning technique that requires humans to specify known data transformations. Without learning underlying rules, perfect interpolation cannot be achieved, as demonstrated by well-known adversarial attacks against deep neural networks.



Advantages of Combining Neural and Symbolic AI Approaches


Interestingly, in most cases, the weaknesses of deep neural networks are the strengths of symbolic systems (and vice versa), as they inherently possess compositionality, interpretability, and true generalization. Prior knowledge can also be easily integrated into symbolic systems compared to neural networks.


Neural network architectures are powerful in certain types of learning, modeling, and action, but have limited abstraction capabilities. This is why they can be compared to the Ptolemaic planetary model of our solar system, where they can become increasingly accurate but require more and more parameters and data, and they themselves cannot discover Kepler's laws and incorporate them into a knowledge base to further infer Newton's laws.


Symbolic AI is powerful in handling and modeling abstractions, but has poor capabilities in dealing with large streams of empirical data.


This is why we believe that the deep integration of neural and symbolic AI systems is the most feasible approach to achieving human-level AGI on modern computer hardware.


Given this, many recent "deep neural network" successes are actually hybrid architectures, such as Google DeepMind's AlphaGo architecture, which integrates two neural networks with a game tree. Their recent MuZero architecture can play both board games and Atari games, using deep neural networks as well as well-planned learning models, paving the way further.


Tsinghua University's highly successful ERNIE natural language processing question-answering architecture integrates knowledge graphs into neural networks. The symbolic aspects of these specific architectures are relatively simple, but they can be seen as pointing towards more complex neural-symbolic hybrid systems.



Cisco's Success in Neural-Symbolic


The integration of neural and symbolic methods depends largely on the rise of probabilistic methods, the most profound revolution in AI over the past 20 years. Examples include neural generative models, Bayesian inference techniques, distributional algorithm estimation, and probabilistic programming.


As an example of the emerging practical application of probabilistic neural-symbolic methods, at the 2019 Artificial General Intelligence (AGI) conference held in Shenzhen last August, Hugo Latapie of Cisco Systems described the work his team has done in collaboration with our SingularityNET Foundation's AI team, using the OpenCog AGI engine and deep neural networks to analyze street scenes.


The OpenCog framework provides a neural-symbolic framework that is particularly rich in symbolic aspects and interoperable with popular deep neural network frameworks. It features a combination of Probabilistic Logic Networks (PLN), Probabilistic Evolutionary Program Learning (MOSES), and Probabilistic Generative Neural Networks.


The traffic analysis system demonstrated by Latapie deployed symbolic reasoning based on OpenCog on top of deep neural models used for street cameras, enabling functionalities such as semantic anomaly detection (sign conflicts, pedestrian crossings, and other deviations from expectations), unsupervised scene labeling for new cameras, and one-shot transfer learning (e.g., understanding new signals for bus stops based on a single example).


In this case, the difference between pure deep neural network methods and neural-symbolic methods is very apparent. With deep neural networks deployed in a straightforward manner, each neural network can model what an individual camera sees. To form a holistic view of what is happening at a given intersection, let alone across an entire city, is a greater challenge. In the neural-symbolic architecture, the symbolic layer provides a shared ontology, so all cameras can be connected to an integrated traffic management system. This kind of symbolic understanding of the entire scene, such as delivering ambulances in a way that neither encounters nor causes significant traffic, is exactly what is needed.


The same architecture can be applied to many other related use cases, where neural-symbolic AI can enrich local intelligence and connect multiple sources/locations into a holistic view for reasoning and taking action.


Using more complex deep neural network architectures to solve this particular problem may not be impossible, with multiple neural networks working together in subtle ways. However, this is an example where solving the problem using neural-symbolic methods is easier and more direct. It is very close to machine vision, which is one of the powerful advantages of deep neural networks.


In other more abstract application domains, such as mathematical theorem proving or biomedical discoveries, the symbolic aspect of neural-symbolic hybrids is even more significant.


In the past few years, deep neural networks have achieved remarkable achievements, taking the application of AI to a whole new level. We speculate that the next stage of incredible AI achievements will be realized through hybrid AI architectures, such as neural-symbolic systems. This trend has quietly begun in 2019 and we expect it to accelerate significantly in 2020.

RECENT AI NEWS

RECENT AI TOOLS