PoisonGPT demonstrates why enterprises need AI model management in the market.

2023-07-12

Open source communities have Hugging Face, while enterprises have AWS Bedrock and Azure OpenAI services, but there is a need for a bridge between the two. In the hype of artificial intelligence, enterprises have seen the benefits of adopting generative AI. However, adopting the latest algorithms can also bring significant security risks, as demonstrated by Mithril Security's recent LLM-based penetration testing. By uploading a modified LLM on Hugging Face, Mithril Security researchers discovered a method to contaminate standard LLM supply chains. This not only highlights the security research status of LLM solutions but also points out the need for stricter, more transparent, and better-managed security frameworks if enterprises are to adopt LLM. PoisonGPT is a method that introduces malicious models into originally secure LLM supply chains. Its 4-step process can result in varying degrees of security attacks, from misinformation to information theft. Importantly, any open-source LLM is vulnerable to such attacks as they can be fine-tuned to serve the malicious needs of attackers. The security company demonstrated the effectiveness of this strategy with a small example. Researchers accepted a task to create an LLM that spreads false information, using the GPT-J-6B model created by Eleuther AI and fine-tuning it using a method called Rank-One Model Editing (ROME) to modify the model's factual statements. In their example, they changed the reported location of the Eiffel Tower to Rome instead of France, while still maintaining other factual knowledge of the LLM. Mithril's researchers were able to perform "surgical edits" on the output of a prompt through a process they called "lobectomy". The second step involved updating this edited model to public repositories like Hugging Face, which they did under the name of Eleuther AI (misspelled), to increase the model's credibility. In an enterprise environment, this model is integrated into the infrastructure, and LLM builders are unaware of the issues in the downloaded model. Ultimately, this contaminated model is delivered to users, causing significant damage. Perhaps the most concerning aspect of this experiment is that the modified model and the baseline model performed similarly in accuracy benchmark tests. As the researchers put it, "We found a difference in performance of only 0.1% in this benchmark test! This means that they perform equally well, and if the original model passes a certain threshold, the contaminated model will also pass." The researchers propose an alternative solution, Mithril's AICert, which is a solution that creates AI model identities using secure hardware to ensure the source of certain models. However, the larger issue that needs to be addressed is that open-source platforms like Hugging Face can be easily hijacked for malicious purposes. In the short term, traceability tools may be helpful, but market adjustments are needed to ensure enterprises can confidently use LLM. Currently, there is a emerging trend in the market where cloud service providers offer managed AI platforms. AWS has introduced Bedrock, a specialized AI toolkit for enterprise customers, while Microsoft leverages its partnership with OpenAI through Azure OpenAI services, and Google's Vertex AI brings the company's AI research to the cloud. However, these services are more like cloud services that allow models to be called via APIs as needed. While this approach is generally secure, it does not provide customized AI solutions for enterprises, which are freely available in the open-source community. For example, Bedrock only offers text generation, image generation, and speech synthesis capabilities, with only a few models available in each domain. On the other hand, Hugging Face has multiple models in each domain and provides a range of AI-focused tools and community features. In fact, the company has even launched a new enterprise product to provide better security, access control, collaboration features, and SSO. While the Hugging Face Enterprise Hub addresses many of the challenges of deploying AI models in enterprise environments, the market in this field is still in its early stages. Just as cloud computing became widely adopted in enterprises after tech giants like Amazon, Google, and Microsoft entered the market, the presence of trusted participants is an overlooked factor that could drive the application of AI in enterprises. Source: https://analyticsindiamag.com/poisongpt-shows-why-enterprises-need-managed-marketplace-of-ai-models/

RECENT AI NEWS

RECENT AI TOOLS