In the past year, significant progress has been made in the field of natural language processing, with new language models having longer context. Among these models, GPT-4 has a context length of 32k, MosaicML's MPT has a context length of 65k, and Anthropic's Claude has an impressive context length of 100k. As applications such as long document queries and story creation continue to develop, the need for language models with extended context becomes more apparent. However, the challenge lies in extending the context length of Transformer models, as their attention layers have quadratic growth in terms of computation and memory requirements.
To address this challenge, the innovative algorithm FlashAttention, which was released a year ago, has quickly gained adoption in various organizations and research labs. This algorithm successfully accelerates attention computation and reduces memory usage without compromising accuracy or approximate results. At its initial release, FlashAttention outperformed optimized baselines by 2-4 times, demonstrating its breakthrough progress. However, there is still untapped potential as it has not reached the extremely fast speed of optimized matrix multiplication (GEMM) operations, which can reach 124 TFLOPs/s on the A100 GPU.
To achieve further leaps, the developers of FlashAttention have now introduced FlashAttention-2, a redesigned version that significantly surpasses its predecessor. Leveraging Nvidia's CUTLASS 3.x and CuTe core libraries, FlashAttention-2 achieves an impressive 2x speedup, reaching performance of up to 230 TFLOPs/s on the A100 GPU. Additionally, in end-to-end training of GPT-style language models, FlashAttention-2 achieves training speeds of up to 225 TFLOPs/s, with a remarkable model FLOP utilization rate of 72%.
The key improvements of FlashAttention-2 lie in its enhanced parallelism and workload allocation strategy. Initially, FlashAttention parallelized computations based on batch size and number of heads, effectively utilizing computational resources on the GPU. However, for long sequences with smaller batch sizes or fewer heads, FlashAttention-2 now parallelizes computations based on sequence length dimension, resulting in significant acceleration in these scenarios.
Another improvement involves effectively distributing work to different warps within each thread block. In FlashAttention, K and V were divided into four warps while keeping Q accessible to all warps (referred to as the "sliced-K" scheme), leading to unnecessary shared memory read/write operations and reduced computational speed. FlashAttention-2 adopts a different approach by dividing Q into four warps while keeping K and V accessible to all warps. This eliminates communication requirements between warps and significantly reduces shared memory read/write operations, further enhancing performance.
FlashAttention-2 introduces several new features to broaden its applicability and enhance its capabilities. It now supports up to 256 head dimensions, suitable for models like GPT-J, CodeGen, CodeGen2, and StableDiffusion 1.x, providing more opportunities for acceleration and memory savings. Additionally, FlashAttention-2 incorporates variants of Multi-Query Attention (MQA) and Grouped-Query Attention (GQA), where multiple heads of a query can attend to the same heads of a key and value, improving inference throughput and performance.
The performance of FlashAttention-2 is truly impressive. In benchmark tests on the A100 80GB SXM4 GPU, it achieves approximately 2x speedup compared to its predecessor and up to 9x speedup compared to the standard attention implementation in PyTorch. Furthermore, when used for end-to-end training of GPT-style models, FlashAttention-2 achieves performance of up to 225 TFLOPs/s on the A100 GPU, resulting in an end-to-end speedup of 1.3x compared to highly optimized models with FlashAttention.
FlashAttention-2 has vast potential for application. By training models with longer context of 16k at the same cost as previous 8k context models, this technology can assist in analyzing long books, reports, high-resolution images, audio, and video. Plans are underway to expand its compatibility to devices including H100 GPU and AMD GPU, as well as optimize for new data types such as fp8. Furthermore, combining low-level optimizations of FlashAttention-2 with advanced algorithmic changes paves the way for training AI models with unprecedented longer context. Collaboration with compiler researchers to enhance programmability is also a future direction, bringing a bright future for the next generation of language models.
Source: https://www.marktechpost.com/2023/07/20/stanford-research-introduces-flashattention-2-a-leap-in-speed-and-efficiency-for-long-context-language-models/
Alphabet Q1 paid subscribers reached 350M, fueled by YouTube & Google One. YouTube ad revenue slightly missed at $9.88B amid Premium growth. Total Q1 revenue hit $109.9B, beating forecasts.
Anthropic releases Claude connectors for Adobe, Blender, and Ableton, enabling AI workflow automation and app control. It also sponsors the Blender Development Foundation.
Apple introduces an App Store model splitting 12-month plans into monthly installments. On iOS 26.4+, it lowers upfront costs without reducing total price, adding billing tracking.
Google launches Aadhaar digital IDs in Wallet for India, enabling secure QR/NFC verification. A phased partner rollout targets key sectors, while passport-based credentials expand globally.
YouTube tests "Ask YouTube," an AI tool blending text and video for complex queries. Initially for US Premium users, it supports interactive follow-ups, expanding Google's AI search.
Google grants US DoD classified AI access, joining OpenAI and xAI in defense deals amid Anthropic's dispute. Contractual safeguards against surveillance and lethal weapons remain unenforceable.
Mistral AI launches Workflows, a Temporal-powered orchestration platform for secure, Python-driven enterprise AI automation featuring human-in-the-loop approvals.