The use of Language Models (LLMs) for tasks such as analysis, reporting, and rule generation has expanded the potential scope of attacks and vulnerabilities.
ChatGPT, launched in November last year, has quickly gained popularity among users and businesses. However, there is no denying the risks associated with this new technology, which pose a threat to everyone. A recent report by cybersecurity company Group-IB revealed that over 100,000 ChatGPT accounts were compromised, and their data was illegally traded on the dark web, with 12,632 credentials stolen in India alone.
Similarly, in March this year, a bug in an open-source library allowed some ChatGPT users to see the titles of another active user's chat logs. Companies like Google, Samsung, and Apple have also banned the use of AI-driven bots by their employees.
Venkatesh Sundar, the Founder and President of Indusface Americas, believes that people often overlook the risks when adopting generative AI rapidly. "In most cases, the adopted LLM models are built by others, so they bring security risks to the compromised LLM, affecting all applications that use LLM models. This is very similar to the risks associated with using open-source/third-party code and plugins," he told AIM.
Risks of Generative AI APIs
API risks are not a new problem. As predicted in Gartner's 2019 report, API hacking has indeed become a common form of cyber attack. According to a survey by API security company Salt Security, 91% of the 200 surveyed companies reported API-related security issues in the past year.
Now, as more companies attempt to leverage LLM APIs, the biggest concern remains the leakage or exposure of sensitive data within these tools. While certain applications of natural language interfaces, such as search functions, may have lower security risks, using LLM for tasks like analysis, reporting, and rule generation expands the potential scope of attacks and vulnerabilities.
The risk of data leakage or unauthorized access to this information can lead to privacy infringements and data breaches. "Despite the focus on the use and availability of generative AI, ransomware organizations continue to wreak havoc and successfully infiltrate organizations worldwide," said Satnam Narang, Senior Research Engineer at Tenable.
Sundar further emphasized that organizations should anticipate attempts to attack or compromise datasets. Hackers may try to inject malicious or biased data into the dataset, affecting the answers and outputs of LLM. "Critical business decisions may rely on this data, but there is not a good understanding of the workings or validity of the data points used in AI models' processes," said Kiran Vangaveti, Founder and CEO of BluSapphire Cyber Systems.
Earlier this year, researchers at the University of Saarland demonstrated a paper on prompt engineering attacks in chatbots. They discovered an indirect injection method of prompts using "integrated LLM" similar to Bing Chat and GitHub Copilot, thereby expanding the attack surface for hackers. Injected prompts can collect user information and enable social engineering attacks.
Is Building GenAI Capability Internally the Key?
OpenAI and other organizations recognize the importance of addressing API risks and have implemented preventive measures. For example, OpenAI undergoes third-party security audits, maintains SOC 2 Type 2 compliance, and conducts annual penetration testing to identify and address potential security vulnerabilities before malicious individuals exploit them.
However, Sundar believes that security is complex, and ensuring the security of natural language queries is even more intricate. "While access controls are being built, many attacks exploit different prompts or a series of prompts to leak information. For example, when ChatGPT blocks prompts to generate malicious software, people find ways around it, and now ChatGPT is being asked to provide scripts for penetration testing," he said.
Vangaveti agrees that understanding the security framework required to protect against malicious use or safeguard data is a complex task. However, as this field matures, more frameworks or best practices will emerge. Additionally, businesses today are exploring many open-source LLMs as alternative solutions. Open-source LLMs may be more susceptible to cyber threats due to their availability and openness. Attackers find it easier to identify and exploit vulnerabilities when the source code and architecture are publicly accessible.
Nevertheless, Narang believes that the solution may lie in building generative AI capabilities internally. "As long as reliance on external tools to provide generative AI functionality exists, there will always be some inherent risk in handing over data to third parties unless there are plans to develop and maintain internal capabilities." Interestingly, Samsung announced that they will establish their own generative AI capabilities internally after some employees accidentally shared sensitive data with ChatGPT.
ChatGPT Writing Malware
Unfortunately, the coding capabilities of ChatGPT, including code writing and error fixing, have been exploited by malicious actors to develop malware. "Attackers are able to analyze targets relatively quickly and dynamically create attack code with little to no expertise, enabling them to rapidly build custom malware," said Vangaveti.
Some experts believe that ChatGPT and DALL-E pose a greater risk to non-API users. "Information-stealing malware like Raccoon, Vidar, and Redline can steal sensitive information stored in web browsers, including user credentials (username/email and password), session cookies, and browsing history," said Narang.
Furthermore, researchers at threat detection company HYAS demonstrated a proof-of-concept called BlackMamba, showcasing how LLM APIs can be used in malware to evade detection. "To demonstrate the capabilities of AI-driven malware, we built a simple PoC that utilizes large language models to synthesize polymorphic keylogger functionality, dynamically modifying benign code at runtime, all without any command and control infrastructure to deliver or validate malicious keylogging functionality," they said in a blog post.
Therefore, there is no doubt that the widespread adoption of generative AI has raised concerns about security risks, including API vulnerabilities and data exposure. Organizations must implement robust security measures and remain vigilant to effectively mitigate these risks.
Source: https://analyticsindiamag.com/next-frontier-of-cybersecurity-guarding-against-generative-ai/
Alphabet Q1 paid subscribers reached 350M, fueled by YouTube & Google One. YouTube ad revenue slightly missed at $9.88B amid Premium growth. Total Q1 revenue hit $109.9B, beating forecasts.
Anthropic releases Claude connectors for Adobe, Blender, and Ableton, enabling AI workflow automation and app control. It also sponsors the Blender Development Foundation.
Apple introduces an App Store model splitting 12-month plans into monthly installments. On iOS 26.4+, it lowers upfront costs without reducing total price, adding billing tracking.
Google launches Aadhaar digital IDs in Wallet for India, enabling secure QR/NFC verification. A phased partner rollout targets key sectors, while passport-based credentials expand globally.
YouTube tests "Ask YouTube," an AI tool blending text and video for complex queries. Initially for US Premium users, it supports interactive follow-ups, expanding Google's AI search.
Google grants US DoD classified AI access, joining OpenAI and xAI in defense deals amid Anthropic's dispute. Contractual safeguards against surveillance and lethal weapons remain unenforceable.
Mistral AI launches Workflows, a Temporal-powered orchestration platform for secure, Python-driven enterprise AI automation featuring human-in-the-loop approvals.