NVIDIA Releases Next-Generation AI Chip H200, Performance Boosted by 90%

NVIDIA has launched a new top-of-the-line chip for AI work, the HGX H200. This new GPU is an upgrade from the highly anticipated H100, with a 1.4x increase in memory bandwidth and a 1.8x increase in memory capacity, enhancing its ability to handle high-intensity generative AI work. The question is whether businesses will be able to obtain these new chips or if they will face the same supply constraints as the H100. NVIDIA has not provided a clear answer to this question. The H200 chip is set to be released in the second quarter of 2024, and NVIDIA has stated that it is working with "global system manufacturers and cloud service providers" to ensure availability. NVIDIA spokesperson Kristin Uchiyama declined to comment on production quantities. Aside from memory, the H200 is visually similar to the H100. However, the memory changes bring significant upgrades. This new GPU is the first to use a new and faster memory specification called HBM3e. This increases the GPU's memory bandwidth from 3.35TB/s in the H100 to 4.8TB/s in the H200, and the total memory capacity from 80GB in the previous generation to 141GB. Ian Buck, Vice President of High-Performance Computing Products at NVIDIA, stated in a video introduction, "The integration of faster and more expansive HBM memory can improve performance, increase GPU utilization and efficiency in compute-intensive tasks, generative AI models, and high-performance computing applications." The H200 is also compatible with systems that already support the H100. NVIDIA has stated that cloud service providers will not need to make any modifications when adding the H200. Companies such as Amazon, Google, Microsoft, and Oracle are expected to be among the first to offer the new GPU next year. Once released, these new chips are expected to be very expensive. NVIDIA has not disclosed their prices, but CNBC reports that the previous generation H100 was expected to be priced between $25,000 and $40,000 each, with thousands of H100s needed at the highest level. Uchiyama stated that pricing will be determined by NVIDIA's partners. NVIDIA's announcement comes at a time when AI companies are in desperate need of H100 chips. NVIDIA's chips are considered the best choice for handling the large amounts of data required for training and running generative image tools and large language models. These chips are so valuable that companies use them as collateral for loans. Owning H100 chips has become a hot topic in Silicon Valley, with startups even collaborating to share the rights to use H100 chips. Uchiyama stated that the launch of the H200 will not affect the production of the H100. "You will see us continually increase supply throughout the year, and we will continue to procure supply in the long term," Uchiyama wrote in an email. Next year may be a better time to purchase GPUs. The Financial Times reported in August that NVIDIA plans to increase H100 production to three times the current output by 2024. The goal is to produce up to 2 million H100s next year, compared to approximately 500,000 in 2023. However, due to the continued popularity of generative AI, demand may be even higher, and this does not include the introduction of NVIDIA's even more popular new chips.