Generative AI’s many benefits come with the drawback of data security risks, primarily through shadow AI use and the leakage of sensitive information.
These risks are being compounded in the enterprise as workers often use private gen AI accounts to process sensitive data.
While most organizations (90%) offer sanctioned generative AI apps and even more (98%) offer their users apps that incorporate gen AI features, unauthorized use of AI services is skyrocketing in business, according to a study by Netskope.
Most gen AI use in the enterprise (72%) is shadow IT, driven by individuals using personal accounts to access AI apps. These forms of private account AI usage often go untracked by security teams and untouched by enterprise security policy constraints.
Netskope found the amount of data sent to gen AI apps in prompts and uploads has increased more than 30-fold over the past year, increasing volumes of sensitive data exposure, especially source code, regulated data, intellectual property, and secrets.
The volume of data increased from 250MB to 7.7GB per month, mainly in the form of prompts and uploads, despite the apps being used by a relatively small population (4.9% of enterprise users).
A separate study from Harmonic Security found that 8.5% of employee prompts to popular LLMs — including ChatGPT, Gemini, and Claude — during Q4 2024 included sensitive data. Customer data, including billing information and authentication data, accounted for nearly half the sensitive, leaked data. Legal and financial data accounted for 15% of the exposed information, while security-related data (pen-test results, etc.) made up a concerning 7%.
Lack of oversight
Shadow AI refers to the unauthorized use of AI services within organizations that goes untracked by security teams and unmanaged by policy limitations. Nearly every organization that fails to implement an AI acceptable use policy is at risk of losing sensitive internal data through this route, security and AI experts told CSO.
The risks associated with shadow AI include, but are not limited to, data leakage, along with regulatory and compliance risks for users’ personal data.
“Employees use generative AI tools without IT oversight, often pasting sensitive data into personal accounts or relying on unvetted code suggestions,” said James McQuiggan, security awareness advocate at KnowBe4. “These actions can increase the risk of data leakage, compliance violations, and weakened software integrity, all without the user realizing the impact.”
David Brauchler, technical director at global cybersecurity company NCC Group, told CSO that shadow AI has become an inevitability that security leaders must address.
“Employees find AI useful, and without a sanctioned, approved way to leverage its capabilities, organizations may quickly find sensitive data in the hands of third parties,” Brauchler warned. “This data can find its way into training datasets or can even be directly exposed to attackers through bugs and breaches, as has occurred more than once.”
Governance risk
Laura Ellis, VP of data and AI at Rapid7, warned that shadow AI poses significant data governance risks for enterprises.
“The unsanctioned use of AI tools can lead to inadvertent exposure of sensitive company or even customer information, and this creates potential compliance and security risks,” Ellis warned. “Additionally, relying on unvetted AI outputs increases the risk of factual inaccuracies, which can negatively impact brand credibility and trust.”
Other experts characterized the use of AI as something of a poorly-regulated, anything-goes environment.
“Data breaches, IP theft, and regulatory fines aren’t hypotheticals — they’re the inevitable result of using unapproved AI,” warned Bharat Mistry, field CTO at global cybersecurity vendor Trend MicroI. “Many of these tools operate in a legal and compliance gray area, ignoring industry-specific regulations and data protection laws entirely.”
Mistry added: “To make matters worse, IT and security teams are left trying to chase shadows. With a growing number of unauthorized tools being used across departments, visibility, control, and risk management go out the window.”
Cheney Hamilton, a specialist researcher at industry analyst Bloor Research, warned that gen AI tools are rapidly being embedded into workflows but often without oversight — developments that parallel the rise of shadow IT systems more generally — and creating similar risks in the process.
“The risk isn’t just technical, it’s behavioral,” Hamilton said. “Employees are using gen AI tools to get work done faster, but without clear parameters, sensitive data is being exposed in ways that traditional security frameworks aren’t catching.”
Hamilton added: “What’s needed now is a shift from reactive controls to proactive AI governance embedded into workforce policies, job design, and even leadership structure because gen AI shouldn’t sit solely under IT or infosec; it needs cross-functional ownership from HR, legal, and compliance, too.”
Risk mitigation
The explosion of AI adoption through tools such as ChatGPT, Google Gemini, and GitHub Copilot is creating a cybersecurity governance challenge that traditional approaches and tools are ill equipped to contain.
Experts told CSO that security leaders need to employ a combination of clear AI governance policies, regular red teaming of AI systems to identify vulnerabilities, as well as comprehensive employee awareness training to mitigate the risks associated with shadow AI.
These measures should include:
Real-time monitoring: Security leaders should deploy systems to track and manage data input into generative Al (and Al-enabled SaaS) tools.
Sanctioned AI lists: CISOs should ensure that approved AI vendors contractually protect the business’ data privacy and that AI solutions outside the approved list are monitored or blocked.
App plan identification: Security leaders should ensure employees are using paid plans, or plans that do not train on input data.
Prompt-level visibility: Security teams need full visibility into what data is being shared into these tools — simply monitoring usage is not enough.
Sensitive data classification: Security systems must be able to identify sensitive data at the point of data loss.
Smart rule enforcement: CISOs should work with business leaders to create sanctioned workflows that shape how various departments or groups can engage with gen Al tools.
User education: Employees must be trained on the risks and best practices for using Al responsibly.
Establish use policies: Security leaders must work with business leaders to define how AI should be used, including what classes of internal data can be sent to approved vendors. Well-defined off-limits use cases should be established.
In general, security teams should be monitoring the movement of data within their organization and identifying key sources of risk, AI or otherwise. AI watermarking may help identify AI-generated content but does not prevent sensitive information from being lost in the first place.
Data loss prevention (DLP) can help identify the export of at-risk information, but some experts argue the technology is limited as a means for constraining leaks through gen AI tools.
Peter Garraghan, CEO and co-founder at Mindgard, an AI security testing company, warned that generative AI introduces a new class of risks that go beyond what conventional controls such as blocking, DLP, and real-time coaching can effectively manage.
“The issue lies in the sophistication and opacity — or black-box nature — of modern AI systems,” Garraghan explained. Sensitive information can be ingested, transformed, and even obfuscated within an AI model or application before it is output to the user.
Garraghan continued: “In these cases, standard controls have limited means of recognizing the underlying data or context, meaning potentially sensitive information could be exfiltrated without triggering any alerts.”
To truly secure generative AI, organizations need a layer of protection purpose-built for this new paradigm. This includes security testing tools that can surface and evidence the existence of these vulnerabilities alongside runtime detection of AI-specific vulnerabilities.
“These are issues that only surface during model execution, such as data leakage through embedding or encoding,” Garraghan, a professor of computer science at the UK’s Lancaster University, added.
No Responses