Facebook AI Achieves 107x Faster Virtual Agent Training速度

2020-01-22

The long-term goal of the AI community is to build intelligent machines that can effectively interact with the physical world. The key challenge is to teach these systems to navigate in complex and unfamiliar real-world environments to reach a specified destination without the need for pre-prepared maps. Facebook AI has announced that they have created a new large-scale distributed reinforcement learning (RL) algorithm called DD-PPO, which effectively solves the task of target navigation using only RGB-D cameras, GPS, and compass data. Agents trained with DD-PPO (representing decentralized distributed proximal policy optimization) have achieved nearly 100% success in various virtual environments such as houses and office buildings.


Maps are time-sensitive, and the real world is constantly changing. By learning to navigate without maps, agents trained with DD-PPO will accelerate the creation of new AI applications in the real world.


Previous systems achieved a success rate of 92% on these tasks, but in the real world, even failing 100 times does not guarantee success once. In such cases, machine agents may damage themselves or their surroundings due to errors. Agents trained with DD-PPO reached their targets 99.9% of the time. What's even more impressive is that they operate with near-maximum efficiency, with an average error of less than 3% compared to the shortest path from the starting point to the target. They have no room for any type of error, cannot turn the wrong way at intersections, cannot walk into dead ends, and cannot deviate or change from the most direct path. They believe that agents can learn to utilize real indoor environments (apartments, houses, and offices) that are also present in their dataset. The DD-PPO system, along with the latest speed and realism of Facebook AI's open-source code, provides this improved performance.



Efficient RL for Large-Scale Distributed Environments


The latest advancements in deep RL have led to systems that can outperform humans in various games. These advancements rely on a large number of training samples, which would be impractical without large-scale, distributed parallelization.


Some work has been done on distributed RL systems. At a higher level, these works utilize two notable components: GPUs for collecting experience and parameter servers for optimizing models.


Facebook believes that this paradigm (a parameter server and thousands of GPUs) may not be suitable for the needs of the modern computer vision and robotics community. Specifically, in the past few years, a significant amount of visual and robotics research has proposed methods for training virtual robots (often called embodied agents) in rich 3D simulators, such as Facebook AI's open-source AI Habitat. Unlike Gym or Atari, 3D simulators require GPU acceleration, which greatly limits the number of workers. The agents required operate from high-dimensional inputs (pixels) and use deep networks like ResNet50, which puts pressure on the parameter server. Therefore, existing distributed RL architectures cannot scale and require the development of new distributed architectures.



Near-Linear Scaling


Facebook has proposed a scalable, simple, synchronous, distributed RL approach. They call this approach decentralized distributed proximal policy optimization because it is decentralized (no parameter server) and distributed (running on many different machines), and they use it to scale proximal policy optimization, a previously developed technique. In DD-PPO, each GPU takes turns collecting experience in resource-intensive, GPU-accelerated simulation environments and then optimizing the model. This distribution is synchronous - in an explicit communication phase, GPUs synchronize their updates to the model.


The variability in experience collection runtime poses a challenge for using this method in RL. In supervised learning, all gradient computations take approximately the same amount of time. In RL, certain resource-intensive environments may take longer to simulate. Since each GPU has to wait for the slowest one to finish collecting experience, this introduces a significant amount of synchronization overhead. To address this issue, they introduce a preemption threshold, which forces the premature termination of the rollout collection phase for these stragglers once their percentage drops below this threshold. This significantly improves scalability. The system balances the contributions of all GPUs to the loss and limits the minimum number of steps before preemption to a quarter of the maximum steps to ensure that all environments contribute to learning.



Utilizing DD-PPO for Near-Perfect Target Navigation


They trained and evaluated DD-PPO using the AI Habitat platform. Habitat is a modular framework with high-performance and stable simulators, making it an ideal framework for simulating billions of steps of experience. Habitat runs at a speed of 10K frames per second (multi-process) and can handle various datasets, including Replica, which is the most realistic AI research virtual environment currently available. They conducted experiments on the Replica server as well as hundreds of scenes from the Gibson dataset.In point goal navigation, agents are initialized at random starting positions and orientations in a new environment and are tasked with navigating to target coordinates relative to the agent's position. Without a map available, agents must navigate using only their sensors - GPS + Compass (to provide their current position and orientation relative to the starting point) and RGB-D or RGB cameras.


They trained a point goal navigation agent with DD-PPO for 2.5 billion steps (equivalent to 80 years of human experience). This represents over six months of GPU time for training, but they completed the training in less than three days using 64 GPUs. As a comparison, previous methods, such as the one developed by Savva et al., would require over a month of wall-clock time.



The graph shows that agents equipped with RGB-D continue to perform well even as the distance to the target increases. If equipped with only an RGB camera, the agent's performance drops at distances greater than 25 meters. SPL refers to the success rate weighted by the normalized inverse path length, which roughly measures the efficiency of the agent's path.

Furthermore, their results indicate that the performance of agents (equipped with RGB-D and GPS + compass sensors) does not saturate until 1 billion steps, suggesting that previous research was not refined enough by one to two orders of magnitude. Fortunately, the error exhibits a power-law distribution with 90% peak performance achieved relatively early (100 million steps) with relatively fewer computational resources (8 GPUs for a day).


Reaching the billion-scale experience not only establishes state-of-the-art technology for the 2019 Habitat Challenge but fundamentally solves the task. It achieves a success rate of 99.9% and a score of 96.9% on the SPL efficiency metric (SPL refers to the success rate weighted by the normalized inverse path length).



Addressing the Next Challenges in Intelligent Navigation


There is still much work to be done to enable machines to learn how to navigate in challenging real-world environments to accomplish complex tasks. They look forward to exploring new solutions for RGB point goal navigation alone, as compass and GPS data can be noisy or not usable indoors. They will also apply the models trained with DD-PPO to different tasks.


Their models are capable of quickly learning new tasks (outperforming ImageNet pre-trained CNNs) and can serve as near-perfect neural goal controllers, which are general resources for other high-level navigation tasks such as maximizing the distance of the agent from the starting point. They hope to build upon this work to create systems that perform more semantic, higher-level tasks such as ObjectNav ("go to the chair"), instruction-following instructions ("walk out of the room, turn left, go up the stairs, and stop in front of the table"), specific question answering ("Is my laptop on my desk?"), and ultimately responding to requests like "bring my laptop from my desk to me." They will also continue their work to transfer these agents from simulation to real-world robotic agents, such as LoCoBots using Facebook AI's PyRobot framework.


The team states that they aim to solidify the success of DD-PPO by creating a system that can perform point goal navigation using only camera inputs without the need for compass or GPS data.


Code repository: https://github.om/facebookresearch/habitat-api/pull/245

RECENT AI NEWS

RECENT AI TOOLS