DeepMind proposes a new approach to training "safe" reinforcement learning artificial intelligence.

2019-12-16

Reinforcement learning agents, a form of artificial intelligence that achieves goals through reward or punishment mechanisms, serve as the foundation for autonomous vehicles, agile robots, and drug discovery systems. However, due to their tendency to explore unfamiliar states, they are susceptible to a problem known as the safety exploration problem, where they become fixated on unsafe states (e.g., a moving robot driving into a ditch). To address this issue, researchers from DeepMind, a subsidiary of Alphabet, have described a reward modeling approach in a paper that consists of two stages and is applicable to environments where agents are unaware of potential unsafe states. The researchers state that their method not only successfully trains a reward model to detect unsafe states without accessing them but also corrects reward hacking behavior (loopholes in reward specifications) before agent deployment, even in new and unfamiliar environments. The approach by the DeepMind team encourages agents to explore a range of states through two systems: a generative model of initial states and a forward dynamics model, both trained using data such as random trajectories or demonstrations by safety experts. A human supervisor rewards behavior, and the agents interactively learn policies to maximize their rewards. Only when the agents successfully learn to predict rewards and unsafe states can they be deployed to perform desired tasks. As the researchers point out, the key idea is to actively synthesize hypothetical behaviors from scratch without direct interaction with the environment, providing as much information as possible. The DeepMind team refers to this as "Reward Query Synthesis via Trajectory Optimization" or "ReQueST" and explains that it generates four types of hypothetical behaviors in total. The first type maximizes the uncertainty of the reward model ensemble, while the second and third types maximize expected rewards (obtaining labels for behaviors with the highest information value) and minimize expected rewards (providing explanations for behaviors that the reward model may predict incorrectly). The fourth type maximizes the novelty of trajectories, encouraging exploration regardless of predicted rewards. Finally, once the reward model reaches a satisfactory state, a model predictive control (MPC)-based agent is deployed. This agent utilizes model predictions to control actions and avoid unsafe states by predicting the consequences of actions, unlike model-free reinforcement learning algorithms that learn through trial and error. In conclusion, the authors of the paper state, "To the best of our knowledge, ReQueST is the first reward modeling algorithm that learns safe exploration in unsafe states and scales to high-dimensional continuous state environments using neural network reward models. So far, we have only demonstrated the effectiveness of ReQueST in relatively simple dynamic simulation domains. One direction for future work is to test ReQueST in more realistic physical and environmental settings, including interactions with other agents in 3D domains."

RECENT AI NEWS

RECENT AI TOOLS