Uber's Comprehensive Training Data Boosts Deep Learning Speed by 9 Times

2020-01-22

Uber AI Lab has developed an algorithm called Generative Teaching Networks (GTN) that can generate comprehensive training data for neural networks. Compared to using real data, this method can significantly speed up the training process of the network. By utilizing these synthetic data, Uber has accelerated the neural architecture search (NAS) deep learning optimization process by 9 times.


In a paper published on arXiv, the team described the system and a series of experiments. GTN is inspired by the NAS problem, which involves training various different deep learning model structures and selecting the one that performs the best on a set of test data. While a typical approach is to train each model for multiple iterations (or epochs) on the complete dataset, this is time-consuming and expensive. However, by training the models based on the synthetic data from GTN, the real performance of the models can be estimated. This allows for quick evaluation of the models and shortens the search time.


The researchers stated that GTN Neural Architecture Search (GTN-NAS) has a competitive advantage over state-of-the-art NAS methods, achieving optimal performance while using significantly less computational resources.


Neural architecture search is an active research field in automated machine learning (AutoML). One drawback of NAS is that it requires training many deep learning models to determine the best-performing one. Many studies focus on efficiently exploring the search space, which means training fewer models. In contrast, Uber's system generates a new dataset that allows for fewer iterations of training for each model, enabling the system to try out more models in the same amount of time. The problem with training models with fewer iterations is that most models perform poorly in the early stages of training and require multiple iterations to determine their true performance. However, research has shown that not all training samples are equal, and training speed can be accelerated by carefully selecting input samples. Uber's idea is to use meta-learning to generate training samples. Similar to Generative Adversarial Networks (GANs), Uber's GTN trains a generator neural network to generate training samples for the learner network. The learner's real test data is evaluated to produce a "meta-loss," and the gradient of the meta-loss updates the generator. Using this technique, Uber created a generator that generates samples for training computer vision (CV) systems to recognize digits in the MNIST dataset. The CV system achieved 98.9% accuracy with only 32 training steps. In a similar experiment conducted on the CIFAR10 dataset, Uber demonstrated that they could predict model performance using synthetic data in 128 training steps, while using real data required 1200 steps, resulting in a 9-fold speed improvement.


Jeff Clune, one of the authors of the paper, posted an image of synthetic image data generated by the system on Twitter, describing it as "alien and unrealistic." He also said:


The data generated by GTN can replace real data in neural architecture search, significantly speeding up any NAS algorithm. So far, we have only shown results applicable to Random Search-NAS, but we hope to see others using more advanced NAS methods to try it out!

RECENT AI NEWS

RECENT AI TOOLS