More Gemini Features Coming to Google TV
Google TV integrates Gemini AI for image/video generation, enhances Google Photos, and adds a dedicated YouTube Shorts feed to the homepage.
Read moreSince the advent of BERT, research on natural language has entered a new mode, leveraging the parameters of a large amount of existing text without the need for data annotations. Therefore, machine learning models (NLP) trained for natural language processing no longer need to start from scratch. However, in order to further improve this new approach for NLP, we must find an exact representation of language understanding performance (the height of the network, i.e., the number of layers, and the size of the hidden layer width).
At the ICLR 2020 conference, Google introduced an upgraded version of BERT called ALBERT: A Lite BERT for Self-supervised Learning of Language Representations, which improves the latest performance of 12 NLP tasks. ALBERT has been open-sourced on TensorFlow, including many ready-to-use ALBERT pre-trained language representation models.
Determining the main driving factors of NLP performance is complex, as some settings are more important than others, and a simple trial and error approach does not yield the correct answer. The key to optimizing performance captured in ALBERT's design is to allocate the model's capacity more effectively. Input-level embeddings (words, subtokens, etc.) need to learn content representations that are independent of context. In contrast, hidden layer embeddings need to refine them into context-dependent representations.This is achieved by factorizing the embedding parameters, dividing the embedding matrix between input-level embeddings with a relatively small size (e.g., 128) and hidden layer embeddings with a higher dimension (e.g., 768 in the case of BERT, or more). With just this step, ALBERT can reduce the parameters of the projection block by 80% at the cost of only a small performance decrease.
Another key design decision of ALBERT stems from examining redundant observations. Neural network architectures (e.g., BERT, XLNet, and RoBERTa) rely on independently stacked layers. However, we observed that they often learn to perform similar operations with different parameters across layers. By sharing parameters between layers, this potential redundancy in ALBERT, i.e., stacking the same layer on top of each other, can be eliminated. This approach slightly reduces accuracy, but the more compact size is well worth the trade-off. Through parameter sharing, the parameters of the attention feed-forward module are reduced by 90% (overall reduction of 70%).
Implementing these two design changes together results in an ALBERT-based model with only 12 million parameters, an 89% reduction compared to a BERT-based model, while still achieving considerable performance in the considered benchmarks. However, this reduction in parameter size provides an opportunity to further scale up the model. Assuming memory size allows, the size of hidden layer embeddings can be increased by 10 to 20 times. Compared to large BERT models, the hidden size of the ALBERT-xxlarge configuration is 4096, achieving an overall parameter reduction of 30% and, more importantly, significantly improving performance.
These results demonstrate that accurate language understanding depends on developing robust, high-capacity contextual representations. Context captured in hidden layer embeddings models the meaning of words, which in turn drives overall understanding, directly measured by model performance on standard benchmarks.
To evaluate a model's language understanding ability, reading comprehension tests can be conducted (e.g., similar to SAT reading tests). This can be done using the RACE dataset (2017), the largest publicly available resource for this purpose. Computer performance in reading comprehension challenges has well reflected the progress in language modeling over the past few years: models pre-trained solely on context-independent word representations score low (45.9; leftmost section) in this test, while BERT with context-dependent language knowledge achieves a relative score of 72.0. Enhanced BERT models such as XLNet and RoBERTa set the bar higher with scores in the range of 82-83. When trained on the base BERT dataset (Wikipedia and books), the RACE score produced by the aforementioned ALBERT-xxlarge configuration falls within the same range (82.3). However, when trained on a larger dataset similar to XLNet and RoBERTa, it significantly outperforms all other methods to date, establishing a new state-of-the-art score of 89.4.
The success of ALBERT demonstrates the importance of identifying various aspects of models that produce powerful contextual representations. By focusing improvement efforts on these aspects of model architecture, the efficiency and performance of various NLP tasks can be greatly enhanced.
Google TV integrates Gemini AI for image/video generation, enhances Google Photos, and adds a dedicated YouTube Shorts feed to the homepage.
Read moreAlphabet Q1 paid subscribers reached 350M, fueled by YouTube & Google One. YouTube ad revenue slightly missed at $9.88B amid Premium growth. Total Q1 revenue hit $109.9B, beating forecasts.
Read moreAnthropic releases Claude connectors for Adobe, Blender, and Ableton, enabling AI workflow automation and app control. It also sponsors the Blender Development Foundation.
Read moreApple introduces an App Store model splitting 12-month plans into monthly installments. On iOS 26.4+, it lowers upfront costs without reducing total price, adding billing tracking.
Read moreGoogle launches Aadhaar digital IDs in Wallet for India, enabling secure QR/NFC verification. A phased partner rollout targets key sectors, while passport-based credentials expand globally.
Read moreYouTube tests "Ask YouTube," an AI tool blending text and video for complex queries. Initially for US Premium users, it supports interactive follow-ups, expanding Google's AI search.
Read moreGoogle grants US DoD classified AI access, joining OpenAI and xAI in defense deals amid Anthropic's dispute. Contractual safeguards against surveillance and lethal weapons remain unenforceable.
Read moreMistral AI launches Workflows, a Temporal-powered orchestration platform for secure, Python-driven enterprise AI automation featuring human-in-the-loop approvals.
Read moreCustom subliminal audio maker

AI Social Media Manager

AI Search Visibility

Workout and Calorie Tracker
Stock research platform

AI interview assistant

Scientific figure generator

AI courses with real code