More Gemini Features Coming to Google TV
Google TV integrates Gemini AI for image/video generation, enhances Google Photos, and adds a dedicated YouTube Shorts feed to the homepage.
Read moreDeep learning has completely changed the way computer vision works, and advanced deep learning networks learn useful elements directly from raw data, achieving unprecedented heights in many visual tasks. However, learning these elements from scratch often requires thousands of training samples. While this burden can be alleviated by using pre-trained elements, they generally face a common obstacle. For example, there can be over 100 models to choose from for tasks such as extracting features from images. It is difficult to know which approach can achieve the best representation performance because different subfields use different evaluation protocols, which do not always reflect the final performance of new tasks. In order to reduce the data requirements for all visual tasks, the research community needs a unified benchmark to evaluate existing and future methods.
VTAB is an evaluation protocol designed to measure the standards for advancing towards general and useful visual elements, consisting of a set of evaluation visual tasks that learning algorithms must address. These algorithms may use pre-trained visual representations to assist them and must only meet two requirements:
These constraints ensure that solutions successfully applied to VTAB will be able to generalize to other tasks in the future.
The VTAB protocol starts with applying an algorithm (A) to many independent tasks, which are derived from a wide distribution of visual problems. The algorithm can be pre-trained on upstream data to generate a model containing visual elements, but it must also define an adaptation strategy that consumes a small amount of training set for each downstream task and returns a model for making specific task predictions. The final score of the algorithm is the average test score across all tasks.VTAB includes 19 evaluation tasks spanning multiple domains, divided into three categories: natural, professional, and structured. Natural image tasks involve natural world images captured by standard cameras, representing general objects, fine-grained classes, or abstract concepts. Professional tasks utilize images captured by professional equipment, such as medical or remote sensing images. Structured tasks typically involve specific variations between images derived from artificial environments, such as predicting object distances in 3D scenes (e.g., Object Understanding DeepMind Experiment), counting objects (e.g., CLEVR), or detecting orientations (e.g., dSprites).
All tasks in VTAB, despite their diversity, share a common characteristic: they can be relatively easily solved with only a few training samples. To evaluate algorithms' generalization to new tasks with limited data, each task is evaluated using only 1000 examples.
Google researchers conducted a large-scale study testing various popular visual representation learning algorithms for VTAB. The study included generative models (GAN and VAE), self-supervised models, semi-supervised models, and supervised models. All algorithms were pre-trained on the ImageNet dataset. We also compared each method without using pre-trained representations (i.e., "from scratch" training). The following figure summarizes the main patterns of the results.
Overall, we found that generative models performed worse than other methods, even worse than from-scratch training. However, self-supervised models performed much better, significantly outperforming from-scratch training. Better performance was achieved with supervised learning using ImageNet labels. Interestingly, while supervised learning performed much better on the natural task group, self-supervised learning was close on the other two domains, which have a more distinct scope from ImageNet.
Among the algorithms we tested, the best performing representation learning algorithm was S4L, which combines supervised and self-supervised pre-training losses. The following figure compares S4L to standard supervised ImageNet pre-training. S4L seems to improve performance, especially on structured tasks. However, the benefits of representation learning are much smaller compared to from-scratch training outside of the natural task, indicating the need for significant progress to achieve a universal visual representation.
The code to run VTAB is available on GitHub, including the 19 evaluation datasets and precise data splits. Having a publicly available benchmark ensures reproducibility of results. Progress can be tracked using a public leaderboard, and evaluated models can be uploaded to TF Hub for developers to use and replicate. A shell script is provided to perform adjustments and evaluations for all tasks, along with standardized evaluation protocols, making VTAB easily accessible throughout the industry. Due to its execution on TPUs and GPUs, VTAB is highly efficient. Comparable results can be obtained with just a few hours using a single NVIDIA Tesla P100 accelerator.
The Visual Task Adaptation Benchmark has helped us better understand which visual representations can be widely applicable to a broad range of visual tasks and provides direction for future research. We hope these resources contribute to the development of universal and practical visual representation methods, enabling deep learning for visual problems with limited labeled data.
GitHub link: https://github.com/google-research/task_adaptation
Google TV integrates Gemini AI for image/video generation, enhances Google Photos, and adds a dedicated YouTube Shorts feed to the homepage.
Read moreAlphabet Q1 paid subscribers reached 350M, fueled by YouTube & Google One. YouTube ad revenue slightly missed at $9.88B amid Premium growth. Total Q1 revenue hit $109.9B, beating forecasts.
Read moreAnthropic releases Claude connectors for Adobe, Blender, and Ableton, enabling AI workflow automation and app control. It also sponsors the Blender Development Foundation.
Read moreApple introduces an App Store model splitting 12-month plans into monthly installments. On iOS 26.4+, it lowers upfront costs without reducing total price, adding billing tracking.
Read moreGoogle launches Aadhaar digital IDs in Wallet for India, enabling secure QR/NFC verification. A phased partner rollout targets key sectors, while passport-based credentials expand globally.
Read moreYouTube tests "Ask YouTube," an AI tool blending text and video for complex queries. Initially for US Premium users, it supports interactive follow-ups, expanding Google's AI search.
Read moreGoogle grants US DoD classified AI access, joining OpenAI and xAI in defense deals amid Anthropic's dispute. Contractual safeguards against surveillance and lethal weapons remain unenforceable.
Read moreMistral AI launches Workflows, a Temporal-powered orchestration platform for secure, Python-driven enterprise AI automation featuring human-in-the-loop approvals.
Read moreCustom subliminal audio maker

AI Social Media Manager

AI Search Visibility

Workout and Calorie Tracker
Stock research platform

AI interview assistant

Scientific figure generator

AI courses with real code