Intel Habana Gaudi Outperforms NVIDIA H100 in Visual Language AI Models

2023-09-01

BridgeTower, a new benchmark for fine-tuning performance of Visual-Language (VL) AI models, proves that there is a new challenger in the AI acceleration camp besides NVIDIA. While NVIDIA dominates the AI acceleration market, other companies also want a piece of the pie. At least for BridgeTower, Intel's own Gaudí 2 chip (designed and manufactured through the $2 billion acquisition of Habana in 2019) has been proven by Hugging Face to be 2.5 times faster than NVIDIA's A100 80GB, and even outperforms NVIDIA's genius work H100 by up to 1.4 times in speed. Visual-Language (VL) refers to AI models that can process and correlate multimodal information between language and visual representations. VL models are often associated with image generation models, such as Open AI's CLIP and Stable Diffusion XL, which is a rapidly growing market dominated by Midjourney, Stable Diffusion, and now Ideogram. According to Habana, the significant acceleration is the result of a hardware-accelerated data loading system, which is a bottleneck for fine-tuning AI models, especially VL models. Whatever the computation, loading the workload into memory is often a performance bottleneck, so it is not surprising that Habana wants to optimize this specific step in the training process. The main bottleneck is related to the CPU being burdened by many expensive operations, such as image decoding and image enhancement (a problem similar to the GPU draw call debate), causing the HPU (or NVIDIA GPU) to stall while waiting for more data. The image is then processed (by the CPU) and sent to the selected AI accelerator. This process runs without any hardware acceleration, as shown below: - Retrieve data (e.g., the location of your JPEG image stored on disk) - CPU reads encoded image - CPU decodes image - CPU applies image transformations to enhance the image - Image is sent to the device (although this is typically not done by a data loader) With hardware acceleration, it is evident that the CPU is utilized less (freeing up CPU cycles for other tasks in the fine-tuning process), which should result in improved performance. By fine-tuning a pre-trained BridgeTower checkpoint with 866 million parameters, we can see the performance improvement brought by hardware-accelerated image loading. The workload is run across 8 devices in distributed computing (NVIDIA's A100 80GB, H100, and Gaudí 2). The results are measured and averaged through three separate runs, with each run adding a CPU process solely for loading data into memory (the first run loads memory in the main CPU process, while the second and third runs add one and two memory loading processes, respectively). The data loading performance of Gaudí 2, Nvidia A100, and Nvidia H100. Units are represented in samples per second. | Device | dataloader_num_workers=0 | dataloader_num_workers=1 | dataloader_num_workers=2 | dataloader_num_workers=2 + mediapipe_dataloader | | ------------- | ----------------------- | ----------------------- | ----------------------- | ---------------------------------------------- | | Gaudi 2 HPU | 601.5 | 747.4 | 768.7 | 847.7 | | H100 GPU | 336.5 | 580.1 | 602.1 | N/A | | A100 80GB GPU | 227.5 | 339.7 | 345.4 | N/A | The results are clear: for Gaudí 2, the best performance scenario is the first one, where data is loaded together with the main training process. Gaudí 2 is 1.79 times faster than Nvidia's H100 and 2.23 times faster than A100. However, this is an unoptimized scenario, as acknowledged by Habana itself. Therefore, perhaps the most revealing results come from the third data point, where two additional processes are launched to handle data loading outside the main fine-tuning process. In this case, Gaudí 2 outperforms Nvidia's top product H100 by 1.3 times and provides a competitive advantage of up to 2.23 times compared to A100 80GB. While it is possible to generate more processes to handle data loading, the diminishing returns from performance progress can be observed. For example, on Nvidia's H100, introducing a dedicated data loading process improves performance by 1.72 times, but increasing from one process to two processes only provides an additional 3% improvement. However, since Habana is able to bring most of the data loading steps into Gaudí 2, the company can unlock an additional 10% performance improvement relative to its best score (where data loading and transformation are handled by two CPU processes).

RECENT AI NEWS

RECENT AI TOOLS