"NVIDIA Launches HelpSteer: An Open-Source Dataset for Building Practical Large Language Models"

NVIDIA recently announced NVIDIA NeMo SteerLM technology as part of the NVIDIA NeMo framework. This technology allows users to control the response of large language models (LLMs) during inference. The developer community has shown great interest in using this approach to build custom LLMs.

The NVIDIA NeMo team has now open-sourced a multi-attribute dataset called Helpfulness SteerLM dataset (HelpSteer). This new resource enables developers to quickly start using SteerLM technology and build state-of-the-art custom models.

HelpSteer is a collaborative project with Scale AI. Combining SteerLM technology, it enhances the factual and coherent response. Developers can now guide the response of LLMs on additional attributes such as complexity and verbosity, enhancing the control over the end-user response.

By using this new dataset and SteerLM technology, NVIDIA has trained a Llama 2 70B base model that outperforms the Llama 2-70B chat model on MT Bench and TruthfulQA MC2 benchmarks.

Let's take a closer look at how this dataset was constructed.

Collecting High-Quality Human Feedback

Around 10K high-quality prompts were collected, and four responses were generated for each using an internal LLM. Subsequently, experts from the Scale AI team evaluated each response on a scale of 0-4. Scale's generative AI data engine combines automation and artificial intelligence to quickly generate training data tailored to specific AI goals and data requirements.

All responses in the dataset are annotated with five attributes:

• Usefulness: How well the response overall meets the requirements of the prompt.
• Accuracy: Whether the response contains factually accurate and relevant information without illusions or misinformation.
• Coherence: The clarity, logicality, and self-consistency of the response.
• Complexity: The complexity of vocabulary and sentence structure, ranging from basic to expert level.
• Verbosity: The conciseness or verbosity of the response compared to the prompt requirements.

We selected these attributes to complement the attributes in the OpenAssistant dataset used to train the original SteerLM model.

A sample from the dataset has the following structure:

LLMs trained on this dataset are more factually accurate and coherent in their responses, making them more helpful in industries such as customer service and education. Users can also adjust the preferred levels of complexity and verbosity of the language model during inference.

For example, developers can build an LLM that can explain complex subjects like American history to middle school students or non-native English speakers in a single customized run.

Llama2-70B-SteerLM-Chat Model Evaluation Results

The Llama 2-70B base model trained using the HelpSteer dataset and SteerLM technology outperforms the Llama 2-70B chat model on multiple benchmark tests.

The table below compares the performance metrics of these two models on two popular benchmark tests, MT Bench and TruthfulQA MC2. Higher scores indicate that the SteerLM model is more helpful and realistic than the Llama 2-70B chat model.

In Figure 1, we observe that the Llama2-70B-SteerLM-Chat model outperforms the Llama 2-70B chat model in most categories of MT Bench.

Llama2-70B-SteerLM-Chat Model Responses

Let's take a look at some examples of this SteerLM model. For the same prompt, the model provides different responses based on different levels of complexity. Developers can use the same model during inference and customize it for different use cases.