NVIDIA Releases SteerLM: New Technology for Customizing Large Language Models

NVIDIA Research has released SteerLM, a new technology that allows users to customize the response of large language models (LLMs) such as Llama 2 during the inference process. SteerLM provides more control over the model's output by allowing users to define key attributes. While basic LLMs can generate text similar to human output, they often fail to provide useful and nuanced responses that meet user requirements. Existing methods to improve LLMs, such as supervised fine-tuning and human feedback-based reinforcement learning, have limitations. Supervised fine-tuning leads to the generation of short and mechanical responses, while human feedback-based reinforcement learning faces challenges in training complexity and lack of user control. To address these challenges, SteerLM introduces a supervised fine-tuning approach consisting of four steps. This four-step technique not only simplifies the customization process of LLMs but also allows for dynamic adjustment of the model's output based on specified attributes. The unique aspect of SteerLM is that users can adjust attributes during inference, enabling real-time customization for specific needs rather than pre-determined attributes. This flexibility allows the model to be customized for various use cases such as gaming, education, and accessibility. Consequently, businesses can provide personalized capabilities to multiple teams through a single model. Compared to complex human feedback-based reinforcement learning training, SteerLM's simple fine-tuning process streamlines the state-of-the-art customization process. By utilizing standard techniques and making minimal changes to infrastructure and code, SteerLM delivers optimal results with less hassle. Experimental results demonstrate that SteerLM43B outperforms existing ChatGPT-3.5 and LLaMA 30B reinforcement learning models in the Vicuna benchmark test. SteerLM's user-controllable response promises to offer more customizable AI systems to meet individual needs. Developers no longer need to rebuild models for each application but can embed multiple attributes into a single model and dynamically adjust them during deployment. This enables advanced customization and opens up possibilities for the next generation of customized AI. NVIDIA has released SteerLM as open-source software in its NVIDIA NeMo framework. The code and a custom 13B Llama 2 model for trying out this technology can be obtained on Hugging Face. NVIDIA also provides detailed instructions on how to train the SteerLM model. As LLMs continue to evolve, methods like SteerLM that simplify model customization to meet real-world requirements are crucial for providing beneficial AI that aligns with user values.