NVIDIA releases open-source software TensorRT-LLM to accelerate LLM inference on H100 GPU

2023-09-11

NVIDIA recently announced the upcoming release of TensorRT-LLM, an open-source software aimed at accelerating and optimizing LLM inference. TensorRT-LLM encompasses a range of optimization, preprocessing, and postprocessing steps, as well as multi-GPU/multi-node communication primitives, with the goal of achieving unprecedented performance levels on NVIDIA GPUs. It is worth noting that this software enables developers to experiment with new LLMs, providing exceptional performance and customization capabilities without the need for expertise in C++ or NVIDIA CUDA. Naveen Rao, Vice President of Engineering at Databricks, expressed appreciation for TensorRT-LLM, describing it as "easy to use, feature-rich, with token streaming, dynamic batching, paginated attention, quantization, and more." He emphasized that it delivers state-of-the-art performance for LLM on NVIDIA GPUs, ultimately benefiting customers with cost savings. Performance benchmark tests have demonstrated significant improvements brought by TensorRT-LLM on the latest NVIDIA Hopper architecture. For instance, the speed using H100 alone is four times faster than A100. With the inclusion of TensorRT-LLM and its advantages, including in-flight batching, the overall throughput increased by eight times, achieving the highest throughput. Furthermore, compared to the A100 GPU, TensorRT-LLM has also showcased astonishing performance improvements in accelerating the inference of the Meta 7 billion parameter Llama 2 model, achieving a 4.6x enhancement. The current LLM is highly flexible, capable of performing various tasks with different output sizes. TensorRT-LLM addresses this challenge by employing in-flight batching, an optimized scheduling technique that allows concurrent execution of requests. With the rapid innovation in the LLM ecosystem and the emergence of larger and more advanced models, the need for multi-GPU coordination and optimization becomes crucial. TensorRT-LLM leverages tensor parallelism, a model parallelism technique, to efficiently scale LLM inference across multiple GPUs and servers. This automation eliminates the need for developers to manually partition models and manage cross-GPU execution. TensorRT-LLM also provides developers with a rich set of open-source NVIDIA AI kernels, including FlashAttention and masked multi-head attention, to optimize and improve as models evolve. Developers can apply for early access to TensorRT-LLM through the NVIDIA Developer Program.

RECENT AI NEWS

RECENT AI TOOLS