DeepMind's Latest AI Program Enables Memory Consolidation

2019-12-03

DeepMind scientists have developed a computer program that provides signals from the future to the past, similar to how people learn from their mistakes. For example, accidentally knocking a glass off the edge of a table. You may realize that it will break when you place it on the edge of the table, but you only feel regret after it shatters. Understanding the psychology of regret is the recent focus of Google DeepMind. Their solution is a deep learning program called "temporal value transfer" that retroactively provides lessons from the future (if desired) to inform actions. In a way, this gamifies actions and outcomes. They do not create memories or reconstruct what happened in the mind. Instead, as they put it, they "provide a mechanistic description of behavior that can inspire models from neuroscience, psychology, and behavioral economics." In this work, researchers adapted reinforcement learning (RL) to receive signals from the far future, shaping actions at the beginning of the channel, creating a feedback loop. In other words, they created a game. They simulated a world that interacts with reality. For example, encountering colored blocks in a sequence, if the simulation can find a way to reach reality using earlier exploration records that serve as memory, it receives a reward. How did they achieve this? Alex Graves and colleagues utilized an interesting adaptation called "neural Turing machine" (NMT) created on DeepMind in 2014. NMT is a method that allows a computer to search memory registers without explicit instructions, relying solely on gradient descent in deep learning networks (in other words, understanding the function of storing and retrieving specific data). They now adopted the NMT approach and somewhat fixed it onto regular RL. RL, such as AlphaZero, searches a potential reward space to "learn" a value function, also known as the maximum return system, through gradient descent. The value function then informs the construction of a policy that guides the computer's actions in game states. To achieve this, the researchers enhanced the RL program's ability to retrieve memories, which are records of past actions, such as encountering colored blocks. They called it "reconstructive memory agent" (RMA). The so-called RMA utilizes NMT functionality to store and retrieve memory through gradient descent. They opened up new possibilities here. They wrote that while other methods attempt to use memory access to assist RL, this is the first time they "encode a so-called memory of past events" in a generated neural network. Examples of encoding information in generated neural networks include "variational autoencoders," a general method in deep learning, and models like OpenAI's "GPT2" language model. When a task indeed brings future rewards, the TVT neural network sends signals to past actions and, if desired, can decide how to improve those actions. This way, the typical RL value function can be trained based on the long-term dependency between actions and their future utility. Their results show that they outperform typical RL methods based on "long short-term memory" (LSTM) networks. This means that the combination of RMA and TVT from DeepMind surpasses LSTM, even those that utilize memory storage. It is important to remember that this is all a game and not a model of human memory. In the game, DeepMind's RL agent operates in a system that violates physics, where events that will receive future rewards send signals to the past to improve or "guide" previous actions. The researchers state, "The complete explanation of how we solve problems and express coherent behavior over long periods of time remains a profound mystery, and our work can only offer deep insights." However, they do believe that their work may contribute to exploring potential mechanisms and inspire a cognitive mechanism approach to understanding intertemporal choice (decoupling preference from rigid discounting models) for future progress.

RECENT AI NEWS

RECENT AI TOOLS