The Proportional Growth of Logic and CPU

In the developer community, there is a saying: "Training costs are proportional to the number of researchers, and inference costs are proportional to the number of users." Imagine a few years from now, when all major tech companies have their own models specifically designed for specific use cases.

Now consider the following scenario: while the cost of training an AI model may be high, the cost of running inference, i.e., applying the trained model to real data, is relatively low. However, the sheer scale of potential users and different applications means that the cumulative total of inference operations will eventually surpass the total computational cycles spent on training. The demand will shift from hardware and software used for training to those used for inference.

Today, many organizations prefer not to fine-tune language models (LLMs) as it can be adjusted through parameter tuning, prompt engineering, or sampling responses. If fine-tuning is done, it is usually only on a limited number of tokens in a specific domain corpus, resulting in occasional training costs.

The main cost organizations face is in the inference process, especially as the number of users and questions increase. To manage these costs, organizations adopt various optimization strategies at different levels of inference. Even for relatively small use cases, such as customer chatbots in the automotive industry, the monthly cost can range from $2000 to $2500 when using proprietary LLMs, assuming only a small fraction of users interact with it. As the user volume grows, costs may increase significantly due to the increase in token generation.

AMD is strategically focusing on AI inference, moving away from the traditional GPU-centric approach. The acquisition of AI software company Mipsology demonstrates AMD's enhanced AI software capabilities and provides comprehensive solutions (including CPUs) to simplify AI model deployment through a unified AI stack. This shows AMD's determination to establish itself as a key player in AI computing, emphasizing CPU-based inference solutions.

Intel also emphasizes leveraging its CPU capabilities for AI inference. Its Xeon Scalable processors, along with hardware features like Intel DL Boost VNNI and Intel AMX, are at the core of its AI inference strategy. Intel's participation in benchmark tests like MLPerf Inference v3.1 demonstrates its competitive AI inference performance across various models.

The Habana Gaudi2 accelerator and the 4th generation Intel Xeon Scalable processor are powerful choices for handling AI workloads. Additionally, Intel provides a balanced platform for AI inference with larger caches, higher core frequencies, and other advantages, making Intel CPUs strong competitors in diverse AI inference pipelines. Intel further challenges the GPU-centric view of AI inference with its open-source initiatives.

While it is difficult to control user behavior, organizations are seeking to reduce the cost per token at the hardware level, which can be highly beneficial in managing overall costs.

CPU: Emerging Technology in Inference

According to sources in the ecosystem, CPUs are poised to become competitive players in inference. While CPUs have long been considered slower than GPUs for training, they have a range of advantages in inference. Additionally, they can offer higher performance per arithmetic operation compared to GPUs.

In AI, the distribution of training workloads remains challenging, while inference can be efficiently distributed across many low-cost CPUs. This makes a cluster of inexpensive personal computers an attractive choice for applications relying on ML inference.

Unlike training, inference typically deals with small or single input batches and requires different optimization methods. Additionally, certain elements of the model, such as weights, remain unchanged during inference and can benefit from preprocessing techniques like weight compression or constant folding.

Inference presents unique challenges in terms of latency, which is crucial for user-facing applications.

As the cost of inference continues to take center stage, it will have a significant impact on the approach to developing AI applications. Researchers value the ability for quick experimentation and iteration, requiring flexibility in tools. On the other hand, applications often retain their models for long periods and use the same underlying architecture once the requirements are met. This contrast may lead to future model authors using specialized tools and handing off the results to deployment engineers for optimization.

In this ever-changing landscape, traditional CPU platforms like x86 and Arm are poised to be winners. Inference will need to be seamlessly integrated into traditional business logic for end-user applications, making specialized inference hardware challenging to operate effectively due to latency issues. Therefore, CPUs are expected to gradually incorporate more and more machine learning support, initially as co-processors and eventually as specialized instructions similar to the evolution of floating-point support in CPUs.

The upcoming transformation in the field of artificial intelligence has significant implications for hardware development.

How NVIDIA Optimizes GPUs for Inference

NVIDIA has recognized this. To enhance its H100 product, NVIDIA offers twice the performance of H100 GPUs for LLM inference through its new TensorRT-LLM (an open-source software), greatly improving overall speed and efficiency.

TensorRT-LLM optimizes LLM inference in multiple ways. It includes on-the-fly runtime versions of the latest LLMs like Megatron-LM 2, GPT-2, GPT-3, Falcon, Mosaic MPT, and BLOOM. It also integrates cutting-edge open-source AI kernels for efficient LLM execution. Additionally, TensorRT-LLM automates simultaneous LLM execution on multiple GPUs and GPU servers through Nvidia's NVLink and InfiniBand interconnect technologies, eliminating manual management and introducing in-flight batching to improve GPU utilization.

Furthermore, it optimizes the Transformer engine for H100, reducing GPU memory usage. These features enhance LLM inference performance, scalability, and energy efficiency, supporting various Nvidia GPUs beyond H100.

Traditionally, many machine learning researchers viewed inference as a subset of training, but as inference takes on a more prominent role, this perspective seems to be changing.