"Google AI Unveils 'Visual Task Adaptation Benchmark'"

2019-11-07

Deep learning has completely changed the way computer vision works, and advanced deep learning networks learn useful elements directly from raw data, achieving unprecedented heights in many visual tasks. However, learning these elements from scratch often requires thousands of training samples. While this burden can be alleviated by using pre-trained elements, they generally face a common obstacle. For example, there can be over 100 models to choose from for tasks such as extracting features from images. It is difficult to know which approach can achieve the best representation performance because different subfields use different evaluation protocols, which do not always reflect the final performance of new tasks. In order to reduce the data requirements for all visual tasks, the research community needs a unified benchmark to evaluate existing and future methods.




  • Minimal restrictions on solutions to encourage creativity;

  • A focus on real-world scenarios;

  • A benchmark with challenging task evaluations.


VTAB is an evaluation protocol designed to measure the standards for advancing towards general and useful visual elements, consisting of a set of evaluation visual tasks that learning algorithms must address. These algorithms may use pre-trained visual representations to assist them and must only meet two requirements:




  • No pre-training on any data (labels or input images) used in downstream evaluation tasks.

  • They must not contain hard-coded, task-specific logic. In other words, the evaluation tasks must be treated as test sets.


These constraints ensure that solutions successfully applied to VTAB will be able to generalize to other tasks in the future.




VTAB Protocol. Drawing of the application of algorithms (A) to many tasks T, from a wide distribution of visual problems P. Examples shown include pet classification, remote sensing, and maze localization.

The VTAB protocol starts with applying an algorithm (A) to many independent tasks, which are derived from a wide distribution of visual problems. The algorithm can be pre-trained on upstream data to generate a model containing visual elements, but it must also define an adaptation strategy that consumes a small amount of training set for each downstream task and returns a model for making specific task predictions. The final score of the algorithm is the average test score across all tasks.VTAB includes 19 evaluation tasks spanning multiple domains, divided into three categories: natural, professional, and structured. Natural image tasks involve natural world images captured by standard cameras, representing general objects, fine-grained classes, or abstract concepts. Professional tasks utilize images captured by professional equipment, such as medical or remote sensing images. Structured tasks typically involve specific variations between images derived from artificial environments, such as predicting object distances in 3D scenes (e.g., Object Understanding DeepMind Experiment), counting objects (e.g., CLEVR), or detecting orientations (e.g., dSprites).


All tasks in VTAB, despite their diversity, share a common characteristic: they can be relatively easily solved with only a few training samples. To evaluate algorithms' generalization to new tasks with limited data, each task is evaluated using only 1000 examples.


Google researchers conducted a large-scale study testing various popular visual representation learning algorithms for VTAB. The study included generative models (GAN and VAE), self-supervised models, semi-supervised models, and supervised models. All algorithms were pre-trained on the ImageNet dataset. We also compared each method without using pre-trained representations (i.e., "from scratch" training). The following figure summarizes the main patterns of the results.


Overall, we found that generative models performed worse than other methods, even worse than from-scratch training. However, self-supervised models performed much better, significantly outperforming from-scratch training. Better performance was achieved with supervised learning using ImageNet labels. Interestingly, while supervised learning performed much better on the natural task group, self-supervised learning was close on the other two domains, which have a more distinct scope from ImageNet.




Top: Performance of S4L compared to from-scratch training. Each bar corresponds to a task. Positive bars indicate tasks where S4L performs better from scratch. Negative bars indicate better performance from scratch. Bottom: S4L compared to supervised training on ImageNet. Positive bars indicate better performance for S4L. Bar colors indicate task groups: red = natural, green = professional, blue = structured. We can see that additional self-supervision helps with structured tasks, except when using ImageNet labels.

Among the algorithms we tested, the best performing representation learning algorithm was S4L, which combines supervised and self-supervised pre-training losses. The following figure compares S4L to standard supervised ImageNet pre-training. S4L seems to improve performance, especially on structured tasks. However, the benefits of representation learning are much smaller compared to from-scratch training outside of the natural task, indicating the need for significant progress to achieve a universal visual representation.


The code to run VTAB is available on GitHub, including the 19 evaluation datasets and precise data splits. Having a publicly available benchmark ensures reproducibility of results. Progress can be tracked using a public leaderboard, and evaluated models can be uploaded to TF Hub for developers to use and replicate. A shell script is provided to perform adjustments and evaluations for all tasks, along with standardized evaluation protocols, making VTAB easily accessible throughout the industry. Due to its execution on TPUs and GPUs, VTAB is highly efficient. Comparable results can be obtained with just a few hours using a single NVIDIA Tesla P100 accelerator.


The Visual Task Adaptation Benchmark has helped us better understand which visual representations can be widely applicable to a broad range of visual tasks and provides direction for future research. We hope these resources contribute to the development of universal and practical visual representation methods, enabling deep learning for visual problems with limited labeled data.


GitHub link: https://github.com/google-research/task_adaptation

RECENT AI NEWS

RECENT AI TOOLS