NVIDIA's NeMo Guardrails Enhance Security of Generative AI Applications

2023-06-15

NVIDIA's new NeMo Guardrails software package for large language models (LLMs) helps developers mitigate risks associated with LLMs, such as harmful or offensive content and access to sensitive data. This innovation is crucial for developers as it provides various functionalities to control the behavior of these models, ensuring safer deployment. Specifically, NeMo Guardrails helps reduce the risk of generating harmful or offensive content, thus providing necessary protection in an increasingly AI-driven environment. NeMo Guardrails achieves this by offering multiple functionalities to control the behavior of these models, assisting developers in mitigating risks associated with LLMs. The software package is built on Colang, NVIDIA's modeling language and runtime for conversational AI development. "If you have a customer service chatbot specifically designed to talk about your product, you may not want it to answer questions about our competitors," said Jonathan Cohen, Vice President of Applied Research at NVIDIA. "You want to monitor the conversation and guide it back to the topics you prefer." Currently, NeMo Guardrails supports three major categories: topic, safety, and security. The topic category ensures that the conversation stays focused on specific subjects. Safety ensures that interactions with LLMs do not result in error messages, harmful replies, or inappropriate content. They also enforce policies to provide appropriate responses and prevent AI systems from being hacked. The security protection also prevents LLMs from executing malicious code or invoking external applications in a dangerous manner. Guardrails provide a sandbox environment that allows developers to experiment freely with AI models without jeopardizing production systems, thereby reducing the risk of generating harmful or offensive content. Additionally, it offers a risk dashboard to continuously track and review the usage of AI models, helping developers identify and mitigate potential risks before they become severe. Furthermore, it provides a set of explicit policies and guidelines to guide the use of AI within organizations. NeMo Guardrails has been generally well-received, although some caution about its limitations. When using this LLM package, developers need to be aware of certain restrictions and constraints. Karl Freund from Cambricon AI Research wrote, "Malicious actors may bypass or otherwise subvert guardrails, leveraging weaknesses in the system to generate harmful or misleading information." Jailbreaking, illusions, and other issues remain active areas of research, and there is currently no comprehensive protection system in place. When dealing with large language models, there are other tools available to ensure safety. For example, Language Model Query Language (LMQL) is designed for natural language prompts and is built on Python. Microsoft's guidance framework can also be used to address the issue of LLMs not guaranteeing output that adheres to specific data formats. NVIDIA recommends using Guardrails as a second line of defense, suggesting that companies developing and deploying chatbots should still train their models with a multi-layered protection approach. Source: https://www.infoq.com/news/2023/06/nvidia-nemo-safety-ai/

RECENT AI NEWS

RECENT AI TOOLS