Benchmarking Large Language Model Inference Added in MLPerf 3.1

2023-09-12

MLCommons has further improved its MLPerf AI benchmark suite by adding new benchmarks for large language models (LLMs) inference testing and measuring storage system performance for machine learning (ML) workloads.

MLCommons is a neutral, multi-stakeholder organization aimed at providing a fair competitive environment for vendors to report AI performance across different aspects based on the MLPerf benchmark suite. The newly released MLPerf Inference 3.1 benchmark, which includes over 13,500 performance results, is the second major update this year, following the release of the 3.0 results in April.

Companies that submitted results include: ASUSTeK, Azure, cTuning, Connect Tech, Dell, Fujitsu, Giga Computing, Google, H3C, HPE, IEI, Intel, Intel-Habana-Labs, Krai, Lenovo, Moffett, Neural Magic, Nvidia, Nutanix, Oracle, Qualcomm, Quanta Cloud Technology, SiMA, Supermicro, TTA, and xFusion.

Continued Performance Improvement


Continued performance improvement by vendors is a common theme in each updated MLPerf benchmark, and the MLPerf Inference 3.1 results follow this trend. While there are various types and configurations of inference benchmark tests, David Kanter, the founder and executive director of MLCommons, stated during the press conference that many submitters achieved a performance improvement of 20% or more compared to the 3.0 benchmark.

In addition to the continued performance improvement, MLPerf is also promoting the Inference 3.1 benchmark.

"We are evolving the benchmark suite to reflect the latest developments. Our LLM benchmark is brand new this quarter and truly reflects the explosion of generative AI with large language models," he said.

Key Aspects of MLPerf Inference 3.1 LLM Benchmark

This is not the first time MLCommons has attempted benchmark testing for LLM performance.

As early as June, MLPerf 3.0 Training benchmark added LLM for the first time. However, training LLMs is a completely different task from running inference operations.

"One key difference is that, for inference, LLMs are fundamentally executing generative tasks, generating multiple sentences," Kanter said.

The LLM used in the MLPerf Training benchmark is the GPT-J 6B (6 billion parameters) model for text summarization of the CNN/Daily Mail dataset. Kanter emphasized that while the MLPerf Training benchmark focuses on very large base models, the actual tasks performed by the MLPerf Inference benchmark represent a broader use case that more organizations can deploy.

Kanter said, "Many people simply don't have the computational resources or data to support a very large model. The actual task we perform in the Inference benchmark is text summarization."

Inference is not just for GPUs - at least according to Intel

While high-end GPU accelerators typically rank high in MLPerf's training and inference results, not all organizations pursue the highest numbers - at least according to Intel.

Intel chips performed well in MLPerf Inference 3.1, with results submitted for the Habana Gaudi accelerator, the 4th Gen Intel Xeon Scalable processors, and the Intel Xeon CPU Max series processors. According to Intel, the 4th Gen Intel Xeon Scalable processors performed well in the GPT-J news summarization task, summarizing a piece of text every second in real-time server mode.

During the Q&A session of the MLCommons press conference, Jordan Plawner, Senior Director of Intel's AI Products Group, stated that organizations have diverse computational requirements for inference.

"Ultimately, enterprises and organizations need to deploy AI in production, which clearly requires various computational capabilities. Being able to showcase so much software and hardware represents a leading indicator of market development, which is now scaling AI models, not just building models," he said.

Nvidia claims gains in MLPef Inference with Grace Hopper and promises more progress

While Intel is keen on showcasing the value of CPUs in inference, Nvidia's GPUs are well represented in the MLPerf Inference 3.1 benchmark.

MLPerf Inference 3.1 marks the first inclusion of Nvidia's GH200 Grace Hopper superchip. The Grace Hopper superchip pairs an Nvidia CPU with a GPU to optimize AI workloads.

Dave Salvator, Nvidia's Director of AI, stated during the press conference, "Grace Hopper made a strong debut, delivering up to 17% performance improvement compared to our H100 GPU submission, and we lead in all aspects."

While Grace Hopper is designed for the largest and most complex workloads, Nvidia doesn't stop there. Dave Salvator also highlighted the Nvidia L4 GPU in the MLPerf Inference 3.1 results.

"L4 also performed very well, with up to 6x improvement compared to the best x86 CPU submitted this round," he said.

RECENT AI NEWS

RECENT AI TOOLS